Technical  Report  1096 


Digital  Procedural  Skill  Retention  for  Selected 
Ml  A2  Tank  Inter-Vehicular  Information  System 
(IVIS)  Tasks 


William  R.  Sanders 

U.S.  Army  Research  Institute 


August  1999 


19990922  027 

United  States  Army  Research  Institute 
for  the  Behavioral  and  Social  Sciences 


Approved  for  public  release;  distribution  is  unlimited. 


DTIC  QUALITY  INSPECTED  4 


U.S.  Army  Research  Institute 

for  the  Behavioral  and  Social  Sciences 

A  Directorate  of  the  U.S.  Total  Army  Personnel  Command 


EDGAR  M.  JOHNSON 
Director 


Technical  Review  by 

Douglas  H.  Macpherson 
Jean  L.  Dyer 
David  W.  Bessemer 


NOTICES 

DISTRIBUTION:  Primary  distribution  of  this  Technical  Report  has  been  made  by  ARI. 
Please  address  correspondence  concerning  distribution  of  reports  to:  U.S.  Army 
Research  Institute  for  the  Behavioral  and  Social  Sciences,  Attn:  TAPC-ARI-PO,  5001 
Eisenhower  Ave.,  Alexandria,  VA  22333-5600. 

FINAL  DISPOSITION:  This  Technical  Report  may  be  destroyed  when  it  is  no  longer 
needed.  Please  do  not  return  it  to  the  U.S.  Army  Research  for  the  Behavioral  and  Social 
Sciences. 

NOTE:  The  findings  in  this  Technical  Report  are  not  to  be  construed  as  an  official 
Department  of  the  Army  position,  unless  so  designated  by  other  authorized  documents. 


REPORT  DOCUMENTATION  PAGE 


1.  REPORT  DATE  (dd-mm-yy)  2.  REPORT  TYPE 

August  1999  Final 

4.  TITLE  AND  SUBTITLE 

Digital  Procedural  Skill  Retention  for  Selected  M1A2  Tank 
Inter-Vehicular  Information  System  (IVIS)  Tasks 


6.  AUTHOR(S) 
William  R.  Sanders 


3.  DATES  COVERED  (from. ..  to) 

September  1996  -  September  1998 


5a  CONTRACT  OR  GRANT  NUMBER 


5b.  PROGRAM  ELEMENT  NUMBER 

0602785A _ 


5c.  PROJECT  NUMBER 

A700  _ 


5d.  TASK  NUMBER 


7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 

U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences 
ATTN:  TAPC-ARI-IK 
2423  Morande  St. 

Fort  Knox,  KY  40121 

9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 
U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333-5600 


12.  DISTRIBUTION/AVAILABILITY  STATEMENT 
Approved  for  public  release;  distribution  is  unlimited. 


5e.  WORK  UNIT  NUMBER 


8.  PERFORMING  ORGANIZATION  REPORT  NUMBER 


10.  MONITOR  ACRONYM 


11.  MONITOR  REPORT  NUMBER 

Technical  Report  1096 


14.  ABSTRACT  (Maximum  200  words): 

The  U  S  Army  Force  XXI  program  makes  extensive  use  of  digital  communications  technologies  to  speed  the  exchange  of 
information  among  all  operational  levels.  While  digital  communications  offers  great  potential,  anecdotal  reports  from  field 
trials  and  testing  repeatedly  state  that  the  basic  procedural  skills  needed  to  operate  these  systems  are  highly  perishable.  The 
present  research  developed  estimates  of  digital  procedural  skill  retention  for  the  tasks  of  creating  and  sending  digital  map 
overlays  and  reports,  using  the  Ml  A2  Abrams  tank  Inter-Vehicular  Information  System.  Twenty-eight  soldiers  received 
instruction  based  on  the  Ml  A2  New  Equipment  Training  Team  lesson  plan,  followed  by  an  immediate  evaluation  of  task 
performance,  and  a  follow-on  evaluation  30  days  later.  Results  showed  a  52  percent  reduction  in  the  number  of  soldiers  able 
to  create  and  send  digital  map  overlays  after  the  30  day  delay,  and  a  23  percent  reduction  in  the  number  able  to  create  and  send 
digital  reports.  Methods  for  measuring  skill  decay  are  presented,  and  an  approach  to  identify  performance  errors  is  provided. 


15.  SUBJECT  TERMS 

Inter- Vehicular  Information  System 

Retention  _ _ 


Skill  Decay 
Digital  Tasks 


M1A2 


16.  REPORT 
Unclassified 


17.  ABSTRACT 
Unclassified 


18.  THIS  PAGE 
Unclassified 


19.  LIMITATION  OF 
ABSTRACT 

Unlimited 


20.  NUMBER 
OF  PAGES 


21.  RESPONSIBLE  PERSON 
(Name  and  Telephone  Number) 

Mr.  William  Sanders,  DSN  464-2613 


Technical  Report  1096 


Digital  Procedural  Skill  Retention  for  Selected 
Ml  A2  Tank  Inter-Vehicular  Information  System 

(IVIS)  Tasks 


William  R.  Sanders 

U.S.  Army  Research  Institute 


Armored  Forces  Research  Unit 
Barbara  A.  Black,  Chief 


U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences 
5001  Eisenhower  Avenue,  Alexandria,  Virginia  22333-5600 


August  1999 


Army  Project  Number  Personnel  Performance  and 

2O262785A790  Training  Technology 


Approved  for  public  release;  distribution  is  unlimited. 


FOREWORD 


As  the  Army  prepares  to  fight  on  the  digital  battlefield, 
modern  digital  communications  systems  are  being  introduced  that 
have  the  potential  to  greatly  increase  the  capability  to  perform 
rapid  and  dispersed  operations.  Along  with  the  increased 
capabilities,  the  new  systems  can  also  place  great  demands  on 
the  soldier's  ability  to  accurately  remember  and  perform  long 
sequences  of  steps  required  to  complete  digital  tasks.  The  goal 
of  the  present  research  was  to  develop  estimates  of  digital 
procedural  skill  retention  for  a  representative  Army  digital 
communications  system,  the  M1A2  Abrams  tank  Inter-Vehicular 
Information  System.  Anecdotal  accounts  from  field  trials  and 
testing  repeatedly  indicate  that  the  basic  procedural  skills 
needed  to  operate  digital  systems  are  highly  perishable,  and 
that  significant  sustainment  training  is  required  to  maintain 
these  digital  skills.  The  present  research  effort  goes  beyond 
anecdotal  descriptions,  measuring  digital  skill  retention  based 
on  a  controlled  set  of  conditions.  Performance  measurement 
strategies  and  criteria  for  measuring  skill  retention  are 
identified. 

The  research  was  performed  by  the  Future  Battlefield 
Conditions  Team  of  the  Fort  Knox  Armored  Forces  Research  Unit 
(AFRU)  of  the  U.S.  Army  Research  Institute  for  the  Behavioral 
and  Social  Sciences  (ARI)  under  Work  Package  2228,  Force  XXI 
Training  Methods  and  Strategies  (FASTTRAIN) .  ARI's  research  is 
supported  by  a  Memorandum  of  Agreement  between  the  U.S.  Army 
Armor  Center  and  Fort  Knox  and  ARI  entitled  "Manpower,  Personnel 
and  Training  Research,  Development,  Test,  and  Evaluation  for  the 
Mounted  Forces,"  dated  16  October,  1995.  This  research  was 
briefed  to  COL  K.  Gunzelman,  Director,  Mounted  Maneuver 
Battlespace  Lab. 


ZITA  M.  SIMUTIS 
Technical  Director 
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DIGITAL  PROCEDURAL  SKILL  RETENTION  FOR  SELECTED  M1A2 
TANK  INTER- VEHICULAR  INFORMATION  SYSTEM  (IVIS)  TASKS 


EXECUTIVE  SUMMARY 


Requirement : 

The  U.S.  Army's  Force  XXI  program  will  field  smaller  yet 
more  lethal  divisions  where  soldiers  will  make  extensive  use  of 
digital  communications  technologies  to  speed  the  exchange  of 
information  among  all  operational  levels.  While  digital 
communications  offer  great  potential,  anecdotal  reports  from 
field  trials  and  testing  repeatedly  indicate  that  the  skills 
needed  to  operate  these  systems  are  highly  perishable.  The 
present  research  addressed  three  basic  requirements.  The  first 
requirement  was  to  develop  empirically-based  estimates  of 
digital  skill  retention  for  selected  M1A2  Abrams  tank  Inter- 
Vehicular  Information  System  (IVIS)  tasks  which  might  be 
generalizable  to  other  current  and  future  systems.  To  meet  this 
requirement  it  was  necessary  to  develop  estimates  of  skill 
acquisition,  retention,  and  recovery.  The  second  requirement 
involved  investigating  the  contributions  of  key  skill  components 
to  task  success,  to  include  procedural  knowledge,  declarative 
knowledge,  and  psychomotor  skills.  The  third  requirement  was  to 
develop  a  performance  assessment  methods  that  could  be  used  by 
training  developers  to  identify  and  address  the  source  of 
recurring  performance  problems. 

Procedure : 

On  Day  1,  28  soldiers  received  instruction  based  on  the 
M1A2  New  Equipment  Training  Team  lesson  plan,  followed  by  an 
immediate  evaluation  consisting  of  four  overlay  and  three  report 
tasks.  Soldiers  successfully  completing  three  or  more  overlays 
were  identified  as  Overlay  Skilled.  Soldiers  completing  two  or 
more  report  tasks  were  identified  as  Report  Skilled. 

Psychomotor  skill  was  assessed  by  presenting  soldiers  with  a 
cursor  positioning  task.  Declarative  knowledge  of  task 
procedures  was  assessed  with  a  multiple  choice  test.  After  a  30 
day  period  without  training  (Day  30)  the  soldiers  were  again 
tested  on  four  overlay,  and  three  report  tasks,  as  well  as 
psychomotor  skill  and  declarative  knowledge  of  the  tasks. 

Several  strategies  for  estimating  skill  acquisition,  retention, 
and  recovery  were  applied  to  the  data.  A  performance  coding 
method  was  developed  to  identify  the  type  and  frequency  of 
performance  errors . 
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Findings : 

The  findings  from  the  present  research  provide  evidence 
that  representative  digital  procedural  skills  will  show 
significant  skill  decay  after  30  days  without  sustainment 
training.  Specifically,  findings  revealed  a  52%  reduction  in 
the  number  of  Overlay  Skilled  soldiers  able  to  meet  the  three  or 
more  successful  overlay  trials  criteria  again  after  the  30  day 
retention  interval.  Results  also  showed  a  23%  reduction  in  the 
number  of  Report  Skilled  soldiers  able  to  meet  the  two  or  more 
successful  report  trials  criteria  again  after  30  days. 
Psychomotor  skill  associated  with  task  performance  did  not  show 
decay  over  30  days.  The  data  failed  to  show  a  significant 
relationship  between  psychomotor  skill  and  Day  30  task  success. 
Declarative  knowledge  of  task  procedures  showed  significant 
decay  after  30  days,  however  this  knowledge  was  recovered  during 
the  Day  30  trials.  Study  findings  showed  a  significant 
relationship  between  declarative  knowledge  of  task  procedures 
and  success  on  Day  30  tasks.  The  description  of  performance 
errors  by  task,  subtask,  and  individual  task  step  allowed  for 
the  identification  of  the  potential  source,  and  possible 
corrective  actions  for  several  performance  problems . 

Utilization  of  Findings: 

The  present  research  provides  a  "mark  on  the  wall"  skill 
retention  estimate  for  representative  M1A2  IVIS  digital 
procedural  tasks,  and  provides  an  approach  that  can  be  used  to 
investigate  other  digital  tasks.  The  detailed  analysis  of 
performance  errors  identified  where  soldiers  had  difficulty 
interpreting  specific  system  interface  cues,  and  also  identified 
where  training  materials  could  be  enhanced  to  stress  common 
error  avoidance  and  error  recovery  procedures.  Detailed 
performance  error  information  such  as  this  could  greatly  aid 
training  developers  in  prioritizing  training  revision  and 
interface  design  needs.  Research  findings  did  not  yield 
evidence  of  psychomotor  skill  decay  for  IVIS  cursor  tracking 
tasks.  Training  developers  should  continue  to  examine  the 
training  benefit  and  cost  effectiveness  of  including  the 
expensive  full  fidelity  IVIS  hand  controller  as  part  of  the 
digital  communications  training  system.  The  present  research 
showed  a  significant  relationship  between  the  seven-item 
multiple  choice  test  of  IVIS  procedures  knowledge  and  hands-on 
task  performance  30  days  later.  This  finding  suggests  that 
digital  skill  retention  might  be  enhanced  by  including  written 
task  knowledge  testing  as  part  of  training. 
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Introduction 


Requirement 

The  U.S.  Army's  Force  XXI  program  will  field  smaller  yet 
more  lethal  divisions  where  soldiers  will  make  extensive  use  of 
digital  communications  technologies  to  speed  the  exchange  of 
information  among  all  operational  levels.  Along  with  the 
increased  capabilities,  the  new  systems  can  also  place  great 
demands  on  the  soldier's  ability  to  accurately  remember  and 
perform  long  sequences  of  step-by-step  procedural  tasks. 
Anecdotal  reports  from  field  trials  and  testing  repeatedly 
indicate  that  the  basic  procedural  skills  needed  to  operate 
digital  communications  systems  are  highly  perishable.  High 
rates  of  skill  decay  could  limit  the  Army's  ability  to  employ 
this  key  communications  technology,  and  could  require  a  major 
investment  in  skill  sustainment  training.  Army  training 
developers  need  an  accurate  estimate  of  digital  procedural  skill 
retention  for  representative  digital  communications  system 
tasks.  They  also  need  a  performance  assessment  method  that  can 
provide  the  detailed  diagnostic  information  necessary  to  build 
and  refine  both  initial  instruction  and  sustainment  training 
programs . 

Goal  of  the  Present  Research 


The  goal  of  the  present  research  was  to  develop  estimates 
of  skill  retention,  and  a  performance  error  coding  method,  for 
representative  digital  communications  system  tasks  using  the 
M1A2  Abrams  tank  Inter-Vehicular  Information  System  (IVIS) .  The 
term  "skill"  is  used  here  to  refer  to  a  wide  variety  of  complex, 
learned  behaviors  which  typically  require  motor  processes  for 
goal  attainment,  while  the  term  "retention"  will  refer  to  the 
maintenance  of  skills  in  the  absence  of  practice  (Schendel  and 
Hagman,  1991) .  The  IVIS  tasks  selected  for  evaluation  were  (1) 
create  and  send  digital  map  overlays,  and  (2)  create  and  send 
digital  reports.  The  digital  overlay  and  report  tasks  represent 
basic  digital  communications  requirements  that  will  be  common 
across  many  current  and  future  systems . 

Training  developers  need  an  accurate  picture  of  task  skill 
requirements  in  order  to  efficiently  match  limited  resources  and 
training  techniques  to  these  demands.  To  meet  this  goal,  the 
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present  research  investigated  the  contributions  of  procedural 
knowledge,  declarative  knowledge,  and  psychomotor  skill  to 
observed  task  success.  Anectdotal  estimates  of  digital  skill 
decay  are  of  limited  value,  as  there  is  typically  no  record  of 
initial  skill  proficiency,  or  any  indication  of  whether  the  lost 
skills  could  be  recovered.  The  present  research  goes  beyond 
this  limitation  by  providing  initial  skill  acquisition  estimates 
as  a  necessary  prerequisite  to  the  investigation  of  skill 
retention.  Likewise,  an  estimate  of  skill  recoverability  was 
developed  to  identify  how  well  soldiers  could  regain  task 
proficiency  with  unaided  practice.  As  a  final  goal  of  the 
present  research,  an  error  coding  approach  was  introduced  that 
identifies  the  specific  location  of  recurring  performance 
problems,  demonstrating  a  performance  measurement  method  that 
could  greatly  assist  training  developers  in  focusing  training 
revision  and  equipment  design  efforts. 

Concern  Over  Highly  Perishable  Digital  Skills 

The  concern  over  the  apparent  rapid  loss  of  digital  skill 
has  been  well  documented  in  military  research,  and  in  soldiers 
reports  of  their  experience  in  field  operations.  In  particular, 
experience  with  the  M1A2  Abrams  tank  I VIS  digital  communications 
systems  suggests  that  digital  skills  are  highly  perishable,  and 
that  research  should  be  directed  to  identify  and  address  initial 
and  sustainment  training  needs .  Recent  ARI  research  conducted 
by  Ford,  Campbell,  and  Cobb  (1998)  investigated  training  issues 
associated  with  digitization,  focusing  on  the  M1A2  training 
program.  The  authors  concluded  that  the  skill  decay 
characteristics  of  military  tasks  represents  a  research  area 
with  high  potential  for  payback,  stating  that  "If  sustainment 
and  retraining  of  tasks  were  based  on  known  factors  and 
conditions  of  decay  and  performance,  the  savings  in  training 
efficiencies  would  be  significant." 

Attending  to  both  the  advantages  and  risks  associated  with 
the  move  to  digitization,  Quinkert  and  Black  (1994)  state  that 
modern  information  technology  is  a  double  edged  sword  that  can 
greatly  enhance  and  improve  combat  effectiveness  or  overload  and 
cripple  unit  operations.  The  authors  report  that  one  of  the  key 
lessons  learned  from  the  1994  Operation  Desert  Hammer  Advanced 
Warfighting  Experiment  (AWE)  was  that  digital  skills  were 
"higher  order"  in  nature,  perished  easily,  and  required  new 
training  techniques  which  go  beyond  knowing  how  to  push  the 
right  "procedural  buttons".  Looking  at  the  total  force,  Salter 
and  Black  (1998)  examined  the  training  and  sustainment  of 
conventional  or  back-up  skills  for  operations  when  digital 
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system  capabilities  become  degraded.  The  authors  conclude  that 
skill  retention  will  be  a  major  issue  as  the  digitized  force 
accommodates  the  competing  training  time  demands  of  new  digital 
tasks,  modified  old  tasks,  and  unmodified  old  tasks. 

Reporting  his  personal  observations  on  the  state  of  digital 
systems  training,  the  Master  Gunner  for  1st  Cavalry  Division 
described  M1A2  skills  as  "exceptionally  perishable"  (Ryan  & 
Iddins,  1999)  .  Likewise,  in  the  Operational  Test  and  Evaluation 
Command  (OPTEC)  follow-on  post  fielding  test  for  M1A2  (U.S. 
OPTEC,  1997),  crews  expressed  the  need  for  more  sustainment 
training  in  the  use  of  the  IVIS  communications  system  at  the 
unit  level,  either  through  hands-on  training  with  the  real 
equipment,  or  through  the  use  of  an  IVIS  training  device  in 
garrison.  Experience  in  AWEs  Desert  Hammer  (U.S.  Army  Armor 
Center,  1994),  and  Focused  Dispatch  (Elliott,  Sanders,  & 

Quinkert ,  1996)  has  established  that  soldiers  can  acquire  the 
skills  needed  to  operate  the  new  generation  of  computer-based 
systems.  However,  the  Focused  Dispatch  Final  Report  again 
concludes  that  "Digital  skills  are  highly  perishable"  (U.S.  Army 
Armor  Center,  1996) .  This  report  states  that  to  simply  maintain 
digital  proficiency  long  enough  to  complete  the  AWE  requires 
that  the  unit  have  an  extensive,  resourced  training  period,  and 
that  these  skills  must  be  sustained  throughout  the  AWE  by 
continued  training  and  stabilization.  Here,  the  call  is  for 
training  development  specifically  designed  to  address  the 
initial  and  sustainment  training  demands  of  highly  perishable 
digital  skills. 

Procedural  Skill  Retention 


A  number  of  task  characteristics  can  be  identified  that  may 
underlie  the  retention  trends  observed  in  Army  field  trials  of 
digital  communications  systems.  The  performance  of  M1A2  IVIS 
overlay  and  report  tasks  involves  procedural  actions  requiring 
sequential  "step-by-step"  thinking  and  discrete  motor  responses 
to  complete  each  task.  While  the  procedural  responses 
themselves  are  usually  easy  to  execute,  it  is  deciding  what 
responses  to  make  and  in  what  sequence  that  pose  the  main 
problems  for  the  learner  (Schendel  and  Hagman,  1991) .  This 
description  of  procedural  tasks  is  quite  useful  in  suggesting 
that  skill  retention  research  needs  to  attend  to  both  cognitive 
and  psychomotor  contributions  to  performance,  and  that  the 
specific  characteristics  of  a  task  that  facilitate  response 
selection  and  sequencing  can  enhance  retention. 
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Cognitive  and  psychomotor  factors.  Cognitive  contributions 
to  task  performance  and  skill  retention  can  be  examined  in  terms 
of  procedural  and  declarative  knowledge  requirements. 

Procedural  knowledge  has  been  described  as  knowing  how  to 
execute  the  procedures  necessary  to  perform  a  given  task  (many 
of  which  have  become  more  rapid  and  automatic  through  practice) , 
such  as  how  to  use  a  typewriter  by  touch,  operate  a  computer,  or 
disassemble  and  reassemble  a  rifle  (Druckman  &  Bjork,  1991) . 

The  ability  to  rapidly  perform  a  short  sequence  of  IVIS  menu 
option  selections  without  consciously  having  to-  think  through 
each  step  might  reflect  procedural  knowledge.  Retention  of 
procedural  knowledge  is  typically  measured  by  the  extent  to 
which  task  procedures  can  be  demonstrated  through  hands-on 
performance,  rather  than  by  the  extent  to  which  they  can  be 
recalled. 

In  contrast  to  procedural  knowledge,  declarative  knowledge 
is  described  as  the  knowledge  of  facts  or  static  information, 
and  is  often  measured  by  tests  of  recall.  In  the  case  of  M1A2 
IVIS  declarative  knowledge  might  include  the  knowledge  of  menu 
organization  and  rules  associated  with  interface  features 
required  to  send  overlays  and  reports.  This  contrast  between 
procedural  and  declarative  knowledge  is  relevant  to  the  present 
investigation  of  digital  procedural  skill  decay.  Declarative 
knowledge  represents  a  deeper  level  of  knowledge  integration, 
often  associated  with  long  term  memory,  than  procedural 
knowledge.  Research  has  shown  that  the  level  of  knowledge  to 
which  tasks  are  initially  trained  (procedural  vs.  declarative) 
can  be  a  predictor  of  long  term  skill  retention  (Druckman  & 
Bjork,  1991) . 

Skilled  performance  will  typically  require  some  element  of 
psychomotor  skill  for  goal  attainment,  such  as  moving  levers,  or 
pushing  buttons.  In  the  case  of  the  M1A2  IVIS,  tasks  are 
composed  of  a  series  of  discrete  motor  responses  with  a  distinct 
beginning  and  end,  such  as  pushing  menu  option  buttons,  or  using 
a  thumb  controller  to  move  a  cursor  to  a  specific  location  on  a 
map  display.  While  Schendel  and  Hagman  (1991)  have  stated  that 
the  psychomotor  skill  component  of  procedural  tasks  is  often 
minimal,  the  investigation  of  digital  procedural  skill  retention 
should  include  an  assessment  of  how  well  soldiers  are  able  to 
acquire  and  retain  this  essential  skill . 

Task  characteristics.  The  specific  characteristics  of  a 
task  that  facilitate  response  selection  and  sequencing  can 
greatly  impact  skill  requirements,  and  thus  retention  of  task 
proficiency.  Digital  system  tasks  differ  in  their  structure 
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from  more  traditional  tasks  and  may  present  different  training 
demands.  Traditional  mechanical  systems  typically  present  the 
operator  with  continuous  motor  tasks  which  logically  flow  from 
one  to  the  next  and  are  thus  resistant  to  forgetting.  In 
contrast,  modern  digital  system  tasks  will  often  require  an 
operator  to  perform  a  series  of  discrete  procedural  steps,  such 
as  computer  menu  options  navigation,  which  researchers  have 
found  to  be  more  susceptible  to  degradation  (Mengelkoch,  Adams  & 
Gainer,  1971) . 

A  review  of  the  literature  specifically  addressing  the 
training  of  computer  skills  (Throne  &  Lickteig,  1997)  also 
suggests  that  the  tasks  associated  with  the  new  generation  of 
Army  digital  systems  may  require  skills  that  are  highly 
perishable.  As  an  example,  examining  the  skill  decay  issue  in  a 
military  context,  research  has  previously  found  that  the  number 
of  procedural  steps  in  a  task  served  as  the  best  predictor  of 
skill  degradation  (Shields,  Goldberg  &  Dressel,  1979) .  While 
not  directly  measuring  skill  retention  over  time,  evidence  from 
the  Combat  Vehicle  Command  and  Control  (CVC2 )  research  program 
(Du  Bois  &  Smith,  1991)  also  suggested  that  there  might  be  a 
rapid  decay  of  skills  required  for  the  operation  of  the  M1A2 
IVIS  equipment. 

A  number  of  task  characteristics  identified  in  retention 
research  have  been  incorporated  into  the  predictive  model  of 
procedural  skill  retention  developed  by  Rose,  Czarnolewski, 
Gragg,  Austin,  Ford,  Doyle,  and  Hagman  (1985) .  Task 
characteristics  incorporated  in  the  model  include  procedure 
cueing,  psychomotor  requirements,  number  of  procedural  steps, 
and  mental  processing  requirements.  The  model  provides  a  ten- 
characteristic  task  rating  method,  the  User's  Decision  Aid, 
which  allows  the  trainer  to  estimate  task  proficiency  levels  at 
a  particular  time  and  project  the  rate  of  proficiency  loss  over 
a  12 -month  retention  interval.  The  User's  Decision  Aid  was 
developed  from,  and  validated  against,  actual  retention  data 
collected  from  soldiers  serving  in  several  Military  Occupational 
Specialty  (MOS)  areas  (63N,  11B,  and  13B) .  While  the  rating 
scale  was  developed  as  a  retention  prediction  tool,  it  might 
also  assist  equipment  and  training  developers  in  identifying 
characteristics  of  tasks  that  might  represent  skill  retention 
risks,  and  the  means  to  reduce  these  risks. 

M1A2  Abrams  Tank  IVIS 


Having  considered  dimensions  of  knowledge  and  task 
characteristics  as  potential  contributors  to  skill  retention. 
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the  specific  equipment  and  training  associated  with  the  M1A2 
IVIS  can  be  reviewed.  The  IVIS  is  a  digital  communications 
system  developed  originally  as  a  battalion  level  asset.  The 
IVIS  improves  command  and  control  capabilities  by  constantly 
exchanging  and  updating  position  location/navigation  data  with 
other  friendly  users.  The  IVIS  provides  the  M1A2  tank  commander 
with  the  capability  for  digital  transmission  of  reports  and 
overlays  between  vehicles.  IVIS  is  a  software  based  system 
implemented  in  the  Commander's  Integrated  Display  (see  Figure 
1) .  The  IVIS  uses  a  radio  interface  unit  coupled  to  the  Single 
Channel  Ground /Airborne  Radio  System  (SINCGARS)  to  transmit  and 
receive  digital  messages.  The  IVIS  tactical  display  is  a  5  X  6 
inch  monochrome  (black  and  orange)  screen,  and  serves  as  the 
commander's  primary  interface  with  IVIS.  The  IVIS  operator 
navigates  through  software  menu  options  and  enters  data  to 
create  overlays  and  reports  using  multifunction  keys,  screen 
mfenus,  a  multifunction  keypad,  and  the  Commander's  Control 
Handle  Assembly  (CCHA)  cursor  controller.  The  IVIS  formatted 
reports  include  Contact,  Spot,  Call  For  Fire,  medical  evacuation 
(MEDEVAC)  reports,  and  others.  The  IVIS  overlays  include 
Operations  1,  Operations  2,  Fire  Support,  Enemy,  Obstacle 
overlays,  and  overlay  updates.  The  IVIS  also  provides  data  for 
display  of  mutual  position/navigation  information.  Every  15 
minutes  or  100  meters  of  tank  movement  IVIS  automatically 
generates  a  position  report  that  is  transmitted  to  other  tanks 
on  the  IVIS  net  (U.S.  Department  of  the  Army,  1995). 

Formal  instruction  on  operating  procedures  for  the  IVIS  is 
typically  provided  as  part  of  training  for  soldiers 
transitioning  from  MlAl  to  M1A2 .  The  current  IVIS  training  is 
provided  by  a  New  Equipment  Training  Team  (NETT)  consisting  of 
military  instructors  and  contractor  personnel  in  a  classroom 
environment  using  Crew  Station  Trainers  (CSTs) .  The  CSTs  are 
computer  workstations  that  provide  all  the  screens  and  menus 
required  to  teach  the  student  selected  M1A2  tank  systems. 
Soldiers  participate  in  a  four-hour  block  of  instruction  on  IVIS 
mission  planning  features,  and  a  four-hour  block  of  instruction 
on  how  to  create  and  send  reports.  After  this  classroom 
training  soldiers  are  expected  to  maintain  proficiency  on  IVIS 
through  hands-on  practice  at  the  unit. 

Issues  Guiding  Research  Design 

The  present  evaluation  of  digital  procedural  skill 
retention  was  designed  to  address  several  issues.  Original 
level  of  learning  was  identified  by  measuring  task  performance 
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Commanders  Control 
Handle  Assembly  (CCHA) 


IVIS  Display  and  Controls 

Figure  1.  The  IVIS  and  CCHA.  Illustrations  from  Operator's 
Manual,  Operator  Controls,  PMCS,  and  Operation  Under  Usual 
Conditions,  Tank,  Combat,  Full-Tracked:  120-MM  gun,  M1A2 
(TM  9-2350-288-10-1),  U.S.  Army  Tank-Automotive  and  Armaments 
Command,  Warren,  MI.,  1995,  p.  1-73. 

at  the  conclusion  of  training.  Task  organization  was  controlled 
by  following  the  structure  and  content  of  the  M1A2  Abrams  tank 
NETT  lesson  plan,  and  a  30  day  retention  span  between  task 
performance  assessment  was  applied  to  all  participants.  The 
IVIS  overlay  and  report  tasks  were  selected  based  on  their  "real 
world"  features,  and  generalizability  to  other  digital 
communication  system  requirements.  Self-report  assessments  were 
made  of  participants  skill  and  knowledge  (particularly  with 
respect  to  computer  skills),  motivation,  and  several 
biographical  factors  such  as  rank  and  education.  In  addition  to 
the  primary  measures  of  procedural  skills,  measures  of 
declarative  knowledge,  and  psychomotor  skills  were  incorporated 
to  capture  cognitive,  skill-based,  and  affective  components  of 
performance. 

Research  Objectives 

The  present  research  was  designed  to  provide  estimates  of 
procedural  skill  retention,  and  a  performance  error  coding 
method,  for  a  representative  digital  communications  system  (M1A2 
IVIS) .  While  there  have  been  numerous  reports  from  field 
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testing  and  field  experiments  describing  digital  skill  decay, 
these  reports  are  anecdotal  and  cannot  account  for  differences 
in  initial  training,  time  since  training,  tasks  performed,  and 
performance  criteria.  Based  on  issues  identified  in  the 
research  literature,  an  assessment  of  IVIS  interface  psychomotor 
skill  retention  and  the  retention  of  declarative  knowledge  of 
IVIS  tasks  were  also  conducted.  The  research  objectives  were 
met  by  addressing  the  six  central  issues  identified  in  Table  1. 

Table  1 

Central  Research  Issues 


1.  Digital  skill  acquisition:  Estimate  IVIS  procedural  skill 
acquisition  during  initial  train-up. 

2.  Digital  skill  retention:  Estimate  IVIS  procedural  skill 
retention  and  recovery  30  days  after  training. 

3.  Digital  skill  subtasks:  Determine  whether  IVIS  procedural 
subtasks  can  provide  diagnostic  information  to  facilitate 
performance  measurement,  feedback,  and  training  development. 

4.  Digital  skill  task  steps:  Determine  whether  error  patterns 
on  IVIS  task  steps  can  provide  diagnostic  information  to 
enhance  performance  measurement,  feedback,  and  training 
development . 

5.  Psychomotor  skill:  Estimate  the  amount  of  psychomotor 
skill  learning  that  occurs  during  train-up,  and  the  retention 
and  recovery  of  this  skill  30  days  after  training. 

6.  Declarative  knowledge:  Estimate  the  amount  of  IVIS  procedure 
declarative  knowledge  learning  that  occurs  during  train-up, 
and  the  retention  and  recovery  of  this  skill  30  days 

after  training. 


Method 


Participants 


Twenty-eight  soldiers  qualified  in  MOS  19K  (Tank  Crewman) 
participated  in  the  study.  The  ranks  of  the  participants 
included  8  E-2,  4  E-3 ,  12  E-4,  5  E-5,  and  1  E-6.  All  soldiers 
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were  assigned  to  Fort  Knox  at  the  time  of  the  study,  in  either 
1st  Squadron,  16th  Cavalry  Regiment,  or  1st  and  2nd  Battalions, 
81st  Regiment,  1st  Armored  Training  Brigade. 

Equipment 


All  training  and  evaluation  sessions  were  conducted  at  the 
Fort  Knox  Mounted  Warfare  Testbed.  A  research  area  15  X  19  feet 
in  size  was  established  with  portable  wall  partitions  to  enclose 
the  four  CST  workstations  and  associated  equipment.  Four  CST 
workstations  produced  by  General  Dynamics  Land  Systems  Division 
were  used  for  training  and  evaluation.  The  CST  consists  of  a 
SUN  SPARC  desk-top  computer  running  version  2.5a  M1A2  IVIS 
software,  20"  color  touchscreen  monitor,  keyboard,  mouse,  and 
CCHA  (see  Figure  2) .  The  CCHA  incorporates  a  thumb  lever  used 
to  control  cursor  movement.  The  mouse  and  keyboard  were  not 
used  in  the  evaluation.  Each  CST  was  located  on  a  separate  30" 
by  60"  table  where  three  soldiers  and  the  instructor  performed 
tasks . 


Cursor  Controller 


Figure  2.  Crew  Station  Trainer.  Illustration  from  M1A2  Crew 
Station  Trainer  (CST)  User's  Manual,  General  Dynamics  Land 
Systems  Division,  Warren,  MI.,  (1997),  p.2. 

Three  CSTs  were  set  up  as  student  stations  with  video 
recording  equipment  to  capture  student  task  performance.  The 
CST  screen  information  presents  all  IVIS  button  pushes  and  menu 
item  selections,  as  buttons  and  menu  items  are  highlighted  or 
change  values  when  selected.  A  constant  readout  of  time  (hours, 
minutes,  and  seconds)  appears  at  the  top  of  the  IVIS  screen  and 
was  also  recorded.  Video  recording  equipment  at  each  student 
station  included  a  VHS  format  video  cassette  recorder  model  BR- 
3200,  a  color  monitor  model  CVM-1271,  and  a  scan  converter  to 
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allow  the  SUN  SPARC  monitor  signal  to  be  recorded  in  VHS  format, 
providing  a  permanent  record  of  actions  for  data  analysis.  One 
CST  was  set  up  as  the  instructor  station.  This  station  had  an 
overhead  projector  connected  to  the  CST  so  that  the  instructor 
could  demonstrate  IVIS  procedures  to  students  by  projecting  his 
computer  screen  display  onto  a  screen  at  the  front  of  the 
training  area.  A  combination  VCR  and  monitor  with  remote 
control  was  used  for  video  data  reduction. 

Materials 


Video  Data  Reduction  Sheets  were  prepared  to  record 
performance  on  IVIS  overlay  and  report  tasks  (see  Appendix  B) . 
The  sheets  were  designed  to  record  overall  task  success  and 
performance  time,  and  task  step  success.  The  Overlay  Video  Data 
Reduction  Sheet  included  four  entry  blanks  used  to  record  times 
for  Overlay  subtasks.  Twenty-three  task  steps  were  identified 
for  the  IVIS  overlay  tasks,  and  19  task  steps  were  identified 
for  the  IVIS  report  tasks.  Identification  of  task  steps  was 
based  on  a  review  of  the  procedures  outlined  in  the  NETT  lesson 
plan,  and  the  specific  requirements  of  the  experimental  tasks. 

A  separate  Video  Data  Reduction  Sheet  was  prepared  for  each  of 
the  14  experimental  tasks,  and  annotated  with  the  specific 
information  requirements  for  that  task,  such  as  type  of  graphic 
to  be  created,  and  location.  The  data  sheets  included  a  space 
for  experimenter  comments,  which  was  used  to  record  participant 
task  performance  that  did  not  conform  to  the  required  task  step 
sequence . 

Three  versions  of  an  IVIS  System  Procedures  Review 
questionnaire  were  prepared  (see  Appendix  C)  for  administration 
at  different  points  in  the  training  and  evaluation  schedule. 

Each  questionnaire  first  presented  ten  questions  addressing  IVIS 
overlay  and  report  task  knowledge.  Seven  of  the  10  questions 
address  task  knowledge  required  to  perform  the  experimental 
tasks  (questions  1,  2,  4,  7,  8,  9,  10),  while  three  questions 
address  task  knowledge  presented  in  training  but  not  required 
for  experimental  task  performance.  Each  questionnaire  contained 
additional  questions  asking  for  specific  information  such  as 
predictions  of  future  performance,  and  reports  of  practice  on 
IVIS  tasks,  that  were  relevant  at  particular  points  in  the 
experiment  evaluation  sequence.  The  IVIS  System  Procedures 
Review  1  included  questions  addressing  training  adequacy,  and 
the  soldier's  estimate  of  his  ability  to  perform  IVIS  tasks  on 
Day  30.  The  IVIS  System  Procedures  Review  2  included  questions 
addressing  training  adequacy,  motivation,  and  whether  the 
soldier  practiced  tasks  during  the  30  day  retention  period.  The 
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IVIS  System  Procedures  Review  3  included  two  questions  on  the 
adequacy  of  training  in  addition  to  the  basic  set  of  ten  IVIS 
task  knowledge  questions. 

IVIS  CST  Exercises 


IVIS  task  cards.  Fourteen  experimental  tasks  were  created 
to  evaluate  soldiers  skills  in  creating  and  sending  IVIS 
overlays  and  reports.  Experimental  tasks  were  based  on  training 
content  taken  from  the  M1A2  NETT  lesson  plan  "Mission  Planning" 
(U.S.  Army  Armor  Center  and  School,  1997),  and  the  lesson  plan 
"Prepare,  Send,  and  Respond  to  Reports"  (U.S.  Army  Armor  Center 
and  School,  1996a) .  Information  describing  each  experimental 
task  was  printed  on  a  5"  X  7"  index  card  for  use  by  participants 
during  the  experiment.  Eight  IVIS  overlay  experimental  tasks 
were  created  which  required  the  soldiers  to  create  and  send  an 
overlay  with  a  multiple  point  graphic,  a  Target  Reference  Point 
(TRP) ,  and  a  six  character  graphic  label.  Six  IVIS  report 
experimental  tasks  were  prepared  which  required  the  soldiers  to 
create  and  send  a  report  with  a  target  coordinate,  two  targets 
(including  target  subtype  and  number) ,  Friendly  Action,  and 
Enemy  Activity  information.  A  description  of  content  for  the 
eight  overlay  tasks  is  provided  as  Appendix  D.  A  description  of 
the  content  for  the  six  report  tasks  is  provided  as  Appendix  E. 
An  example  of  an  IVIS  overlay  task  card  (reduced  in  size)  is 
provided  as  Figure  3.  An  example  of  an  IVIS  report  task  card  is 
provided  as  Figure  4 . 


UPDATE  AND  SEND  OBSTACLE  OVERLAY 

Update  and  send  your  Obstacle  Overlay  with 
the  following  information  to  your  CO  CMDR 

Using  the  THUMB  CURSOR  insert  a  MINEFIELD 
graphic 

-  Coord  1:  2400  2550 

-  Coord  2:  2400  2500 

Using  the  IVIS  KEYPAD  insert  a  TARGET 
REFERENCE  POINT  graphic 

-  Coord:  2300  2750 

-  Label:  AA0001 

PRESS  THE  DIA6  MODE  BUTTON  WHEN  DONE  (B1A3) 


Figure  3.  IVIS  Overlay  task  card. 
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SEND  SPOT  REPORT 

Report  to  your  CO  CMDR  that  you  see  2  BMP 
armored  personnel  carriers  and  4  T72  tanks. 
Using  THUMB  CURSOR  enter  target  grid 
location  16R  ES  2650  2500.  Report  that  the 
enemy  is  withdrawing  and  you  are  continuing 
with  your  mission. 


PRESS  THE  DIAG  MODE  BUTTON  WHEN  DONE  (A2B4) 


Figure  4.  IVIS  Report  task  card. 

Criteria  for  scoring  trials  as  successful.  The  criteria 
for  successful  task  completion  were  developed  to  match  the 
requirements  for  sending  an  accurate  IVIS  message  in  a  tactical 
environment .  No  performance  time  criteria  for  task  success  were 
established  for  the  IVIS  overlay  and  report  tasks,  due  in  part 
to  the  absence  of  performance  time  criteria  in  the  M1A2  NETT 
lesson  plan.  The  decision  not  to  introduce  a  performance  time 
standard  was  also  made  so  that  soldiers  would  be  less  likely  to 
give  up  on  a  task  during  the  Day  1  evaluation,  and  so  that  Day 
30  task  performance  would  better  reflect  soldiers  ability  to 
recover  skills  by  working  through  the  tasks.  Specific 
requirements  for  each  task  to  be  judged  successful  are  provided 
as  Appendix  F. 

Procedure 


All  soldiers  attended  a  train-up  and  evaluation  session 
(Day  1) ,  and  returned  for  a  follow-on  evaluation  of  IVIS  skills 
30  days  later  (Day  30).  An  outline  of  Day  1  and  Day  30  training 
and  evaluation  events  is  provided  as  Table  2.  The  researcher 
first  described  the  capabilities  of  the  M1A2  tank  IVIS  system, 
and  provided  basic  instruction  on  map  overlays,  and  the  use  of 
the  eight-digit  Military  Grid  Reference  System  (MGRS)  to  find 
the  location  of  points  on  the  IVIS  map  display  (see  Appendix  G) . 
Soldiers  next  received  a  one-hour  introduction  to  IVIS  screen 
information,  menu  structure,  and  the  keypad  and  CCHA  interface. 
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This  was  followed  by  a  one-hour  block  of  instruction  on  IVIS 
reports,  and  a  two-hour  block  of  instruction  on  IVIS  overlays. 

Table  2 

Schedule  Of  Experiment  Events 


Day  1  Session  Events 

1 .  Privacy  Act  Statement 

2 .  Introduction  and  MGRS  basics 

3 .  Demographic  Survey 

4.  Training:  IVIS  screen  information  and  interface 

5.  Psychomotor  skill  evaluation  1 

6.  Training:  IVIS  reports  and  overlays 

7 .  Digital  Skills  Training  Factors  (motivation)  questionnaire 

8.  Participants  perform  seven  IVIS  tasks 


Task  1 

Overlay 

Task  2 

Report 

Task  3 

Overlay 

Task  4 

Report 

Task  5 

Overlay 

Task  6 

Report 

Task  7 

Overlay 

9.  Psychomotor  skill  evaluation  2 

10.  IVIS  System  Procedures  Review  1  (task  knowledge) 


Break  for  30  days 


Day  30  Session  Events 


11.  Introduction  and  MGRS  basics 

12 .  IVIS  System  Procedures  Review  2 


Psychomotor  skill  evaluation  3 
Participants  perform  seven  IVIS 

Task  1 

Overlay 

Task  2 

Report 

Task  3 

Overlay 

Task  4 

Report 

Task  5 

Overlay 

Task  6 

Report 

Task  7 

Overlay 

15.  Psychomotor  skill  evaluation  4 

16.  IVIS  System  Procedures  Review  3 


(task  knowledge) 
tasks 


(task  knowledge) 
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The  IVIS  report  and  overlay  task  training  required  soldiers 
to  create  and  send  reports  and  overlays  with  information  and 
graphics  that  would  later  appear  in  the  experimental  tasks. 
Overlay  and  Report  instruction  included  one  practice  trial  using 
data  presented  on  the  3"  X  5"  IVIS  task  cards  to  familiarize 
soldiers  with  the  way  information  was  formatted  on  the 
experimental  task.  Soldiers  were  encouraged  to  ask  questions  to 
ensure  that  they  understood  the  task  requirements  presented  on 
the  card.  The  IVIS  training  closely  followed  the  organization 
and  content  of  the  M1A2  NETT  lesson  plan  materials  for  mission 
planning,  and  creating  and  sending  reports  (M1A2  New  Equipment 
Training  Team  (NETT)  lesson  plan  "Mission  Planning"  (U.S.  Army 
Armor  Center  and  School,  1997),  and  the  lesson  plan  "Prepare, 
Send,  and  Respond  to  Reports"  (U.S.  Army  Armor  Center  and 
School,  1996a) .  The  hands-on  training  lecture  was  delivered  by 
a  former  Army  IVIS  instructor  with  three  years  experience 
training  soldiers  on  the  use  of  the  IVIS  system. 

Prior  to  beginning  the  set  of  seven  tasks  on  Day  1  and  Day 
30  the  researchers  read  the  following  statement  to  provide  a  set 
of  performance  criteria  for  the  experiment:  "Report  accurate 
information  as  quickly  as  possible.  First  priority  is  accuracy. 
Perform  the  task  as  quickly  as  you  can  without  making  errors." 
This  standard  was  based  on  information  contained  in  Fort  Knox 
Tank  Platoon  Standard  Operating  Procedure  (SOP)  Report 
Guidelines  (FKSM  17-15-3,  1996b) .  The  desire  was  to  have 
soldiers  complete  all  tasks  start  to  finish,  and  that  soldiers 
not  give  up  because  they  perceived  that  a  task  had  taken  too 
long  to  perform. 

Immediately  following  the  Day  1  instruction,  soldiers 
performed  seven  IVIS  tasks.  Because  evaluation  trials  also 
serve  to  provide  the  soldier  with  practice  that  can  improve  task 
performance,  the  number  and  alternating  sequence  of  overlay  and 
report  evaluation  trials  had  to  be  strictly  controlled. 

Soldiers  returned  to  the  test  site  30  days  later  for  a  follow-on 
evaluation  of  IVIS  skills  and  again  performed  seven  IVIS  report 
and  overlay  tasks.  Soldiers  performed  eight  IVIS  overlay  tasks 
and  six  IVIS  report  tasks  in  total.  The  unequal  number  of  tasks 
was  chosen  so  that  an  overlay  task  would  be  the  first  and  last 
trial  in  both  train-up  Day  1  and  follow-on  evaluation  Day  30. 

In  this  way  a  direct  comparison  could  be  made  between  IVIS 
overlay  task  performance  on  the  last  Day  1  overlay  task,  and  the 
first  Day  30  overlay  task.  The  assignment  of  IVIS  overlay  and 
report  tasks  within  the  alternating  sequence  of  tasks  was 
counter -balanced  using  a  Latin  Squares  design.  Questionnaires 
were  administered  to  assess  soldier  knowledge  of  IVIS 
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procedures,  and  motivation.  A  digit  tracking  task  was 
administered  four  times  to  assess  soldier  psychomotor  skills. 

The  tracking  task  required  soldiers  to  use  the  CCHA  to  click  on 
and  highlight  ten  numbers  in  sequence  that  were  presented  on  the 
CST  screen.  The  pattern  and  sequence  of  the  ten  numbers  are 
provided  in  Appendix  H. 

Demographic  Measures 

All  soldiers  completed  a  demographic  questionnaire.  The 
demographic  measures  included:  Rank/Grade,  age,  time  in  the 
military,  duty  position,  experience  with  computers,  and  highest 
level  of  civilian  education.  The  demographic  questionnaire  and 
summary  of  the  responses  are  provided  as  Appendix  I  and  J 
respectively.  One  soldier  was  reassigned  after  completing  Day  1 
training  and  evaluation,  and  did  not  return  for  the  Day  30 
evaluation. 

Performance  Estimation 


Performance  estimates.  In  making  estimates  of  skill  decay 
it  was  first  necessary  to  establish  that  soldiers  were  initially 
trained  to  proficiency  on  Day  1.  The  criteria  for  overlay  skill 
acquisition  was  set  at  successful  completion  of  at  least  three 
of  the  four  overlay  tasks.  Criteria  for  report  skill 
acquisition  was  set  as  successful  completion  of  at  least  two  of 
the  three  report  tasks.  One  estimate  of  learning,  three 
estimates  of  skill  retention,  and  one  estimate  of  skill  recovery 
we  re  deve loped: 

-  Learning:  A  comparison  of  performance  on  First  and  Last 
Day  1  trials  provided  an  estimate  of  skill  improvement  with 
hands-on  practice  immediately  after  the  training  lectures. 

-  Retention  (no  practice) :  A  comparison  of  performance  on 
the  Last  Day  1  and  First  Day  30  trials  provided  an  estimate  of 
skill  retention  over  30  days  without  practice. 

-  Retention  (with  practice) :  A  comparison  of  performance 
on  the  Last  Day  1  and  Last  Day  30  trials  provided  an  estimate  of 
skill  retention  after  the  soldier  had  additional  practice 
completing  the  evaluation  trials  on  Day  30. 

-  Retention  (skill  acquisition  criteria) :  Application  of 
the  Day  1  criteria  for  skill  acquisition  to  Day  30  performance. 
Retention  of  Overlay  Skilled  status  required  successful 
completion  of  at  least  three  of  the  four  Day  30  overlay  tasks. 
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Criteria  for  retention  of  Report  Skilled  status  required 
successful  completion  of  at  least  two  of  the  three  Day  30  report 
tasks.  This  estimate  is  only  appropriate  for  procedural  skill 
assessment,  and  does  not  apply  to  declarative  or  psychomotor 
skills . 

-  Recovery:  A  comparison  of  performance  on  the  First  Day 
30  Trial  and  Last  Day  30  Trials  provided  an  estimate  of 
performance  improvement  across  the  seven  Day  30  evaluation 
trials . 

Performance  data.  The  principal  performance  data  for  this 
research  were  task  accuracy  and  task  completion  time.  Task 
accuracy  was  analyzed  at  three  levels  of  detail:  overall  task 
success,  subtask  success,  and  individual  task  step  success  based 
on  a  review  of  Video  Data  Reduction  Sheets  information.  Times 
were  recorded  for  overall  task  completion,  and  also  for  IVIS 
overlay  sub-task  performance.  Report  subtask  times  were  not 
recorded  as  soldiers  frequently  performed  subtasks  out  of 
sequence,  and  several  subtasks  typically  involved  very  short 
performance  times . 

The  primary  source  of  procedural  performance  data  for  the 
overlay  and  reports  tasks  was  the  videotaped  "screen  capture" 
record  of  soldier  button  pushes  and  cursor  movement  used  to 
perform  each  task.  The  clock  component  of  the  IVIS  screen 
recording  provided  a  continuously  running  record  of  task 
performance  time  in  hours,  minutes,  and  seconds.  The  author  and 
principle  instructor  served  as  video  data  reducers,  viewing  each 
tape  recorded  performance  trial,  and  recording  time  data  and 
performance  actions  using  Video  Data  Reduction  Sheets  developed 
for  this  purpose  (see  Appendix  B) .  Performance  times  and  task 
performance  actions  scoring  required  the  agreement  of  both  video 
data  reducers . 

Statistical  analysis.  The  present  experiment  followed  a 
pair-wise  comparisons  approach  between  selected  Day  1  and  Day  30 
trial  success  rates.  Given  the  binomial  Go/No  Go  nature  of  the 
task  success  data,  a  test  for  correlated  proportions  (McNemar, 
1975)  was  used  to  compare  selected  Day  1  and  Day  30  trials. 
Paired-samples  t-tests  were  used  for  task  performance  time 
comparisons,  and  psychomotor  trial  time  comparisons,  with  alpha 
set  at  0.05. 

Sample  size  power  estimation.  The  number  of  soldiers 
required  for  the  experiment  was  estimated  based  on  power 
estimates  for  task  performance  time  paired  t-test  comparisons. 
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Boldovici  and  Kolasinski  (1997)  identify  0.80  as  the  commonly 
accepted  value  for  power,  and  point  out  that  by  choosing  to 
detect  a  test-retest  performance  time  difference  equal  in  size 
to  one  standard  deviation  in  performance  time,  the  critical  test 
statistic  "d"  has  a  value  of  1.  With  a  =  0.05,  d  =  1.00,  power 
=  0.80,  n  (troop  support)  =  17.  Twenty-seven  overlay  task 
trials  were  performed  over  a  period  of  several  months  while 
refining  the  experimental  tasks.  Analysis  of  these  early  trials 
yielded  a  mean  performance  time  of  167.22  seconds  for  overlay 
tasks,  with  a  standard  deviation  of  70.36  seconds.  A  time  value 
equal  to  the  standard  deviation  was  accepted  as  a  reasonable 
performance  effect  to  detect  in  comparing  overlay  trials. 

Results 


Results  Overview 

IVIS  Overlay  procedural  skill  retention.  The  central 
findings  for  the  present  research  are  summarized  in  Table  3.  On 
Day  1,  22  soldiers  successfully  completed  three  or  more  of  the 
four  overlay  tasks.  This  group  constituted  the  100  percent 
Overlay  Skilled  sample  for  follow-on  evaluations  of  skill 
retention.  On  Day  30  only  48  percent  of  the  Overlay  Skilled 
group  was  able  to  successfully  complete  three  or  more  of  the 
four  overlay  trials,  representing  a  significant  52  percent 
reduction  in  the  proportion  of  Overlay  Skilled  soldiers  after  30 
days . 


The  comparison  of  success  rates  on  individual  trials 
provided  evidence  that  overlay  task  performance  decreased 
significantly  over  the  30  day  retention  interval  and  was  not 
recovered  with  practice  on  Day  30.  The  proportion  of  Overlay 
Skilled  soldiers  successfully  completing  overlay  tasks  dropped 
significantly  from  86  percent  on  the  Last  Day  1  trial  to  only  29 
percent  on  the  First  Day  30  trial  (a  decrease  of  57  percentage 
points) .  On  the  final  Day  30  trial  the  overlay  task  success 
rate  had  only  risen  to  48  percent,  so  that  a  significant  38 
percentage  point  decrease  in  skills  remained  compared  to  the 
Last  Day  1  trial  success  rate  of  86  percent. 

The  comparison  of  overlay  performance  times  for  successful 
trials  revealed  significant  improvement  across  Day  1  trials, 
with  performance  times  decreasing  an  average  of  34  seconds  from 
first  to  last  trial.  The  significant  drop  in  performance  time 
provided  evidence  of  continued  learning  with  practice  on  Day  1, 
after  the  formal  training  lecture  had  been  completed.  Overlay 
mean  performance  time  increased  by  321  seconds  across  the  30  day 
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retention  interval  providing  evidence  of  significant  skill 
decay.  Day  30  trials  performance  time  did  show  significant 
recovery  of  skill  with  practice,  with  mean  time  to  perform  tasks 
decreasing  by  315  seconds.  Across  Day  30  Overlay  trials 
performance  times  decreased  to  the  point  where  there  was  no 
evidence  that  Last  Day  30  trial  times  differed  significantly 
from  the  Last  Day  1  times . 

Table  3 


Summary  of  Digital  Skill  Retention  Comparisons 


Skill 

Criteria 

Retention 

Without  Practice 
(Last  Day  1  vs. 
First  Day  30) 

Retention 

With  Practice 
(Last  Day  1  vs. 

Last  Day  30) 

Overlay  Task 

Success  Rate 

57%***  Lower 

38%*  Lower 

Overlay  Task 
Performance  Time 

321*  Seconds 
slower 

No  significant 
difference 

Overlay  Skilled 
Criteria 

Not  applicable 

52%***  Fewer  skilled 

Report  Task 

Success  Rate 

No  significant 
difference 

No  significant 
difference 

Report  Task 
Performance  Time 

52***  Seconds 
slower 

No  significant 
difference 

Report  Skilled 
Criteria 

Not  applicable 

23%**  Fewer  skilled 

Psychomotor 

Task  Time 

No  significant 
difference 

No  significant 
difference 

Declarative 
Knowledge  Score 

29%***  Lower 
test  score 

No  significant 
difference 

Note.  Overlay  results  are  for  the  Overlay  Skilled  sample.  Day  1 
n  =  22,  Day  30  n  =  21.  Report  results  are  for  the  Report 
Skilled  sample,  Day  1  n  =  23,  Day  30  n  =  22. 

*  ^  <  . 05 .  **  p  <  .01.  ***  2  <  -001. 
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I VIS  Report  procedural  skill  retention.  On  Day  1,  23 
soldiers  successfully  completed  two  or  more  of  the  three  report 
tasks.  This  group  constituted  the  100  percent  Report  Skilled 
sample  for  follow-on  evaluations  of  skill  retention.  On  Day  30 
only  77  percent  of  the  Report  Skilled  group  was  able  to 
successfully  complete  two  or  more  of  the  three  report  trials, 
representing  a  significant  23  percent  reduction  in  the 
proportion  of  Overlay  Skilled  soldiers  after  thirty  days.  The 
comparison  of  success  rates  for  individual  trials  failed  to 
provide  evidence  of  additional  skill  learning  on  Day  1  after 
training,  or  of  skill  recovery  on  Day  30  after  completing  the 
evaluation  trials . 

In  contrast  to  the  task  success  criteria,  the  task  time 
criteria  did  reveal  a  significant  46  second  improvement  with 
practice  (learning)  across  Day  1  trials.  Also,  while  report 
success  rate  did  not  reveal  skill  decay  across  the  30  day 
retention  interval,  the  significant  52  second  increase  in  report 
performance  time  did  provide  evidence  of  skill  decay  across  the 
retention  interval.  Across  Day  30  trials  performance  times 
decreased  to  the  point  where  there  was  no  evidence  that  Last  Day 
30  report  trial  times  differed  significantly  from  the  Last  Day  1 
times . 

IVIS  Psychomotor  skill  retention.  Performance  times  for 
the  psychomotor  number  tracking  task  showed  a  significant  23 
second  improvement  across  Day  1  trials.  This  finding  provided 
evidence  that  soldiers  made  significant  improvement  in  their  use 
of  the  CCHA  cursor  controller  with  practice  during  the  seven  Day 
1  trials.  The  time  required  to  complete  the  number  tracking 
task  increased  by  only  three  seconds  over  the  30  day  retention 
interval  which  failed  to  provide  evidence  of  significant  skill 
decay.  The  five  second  improvement  in  performance  time  across 
Day  30  trials  was  small,  but  did  provide  evidence  of 
statistically  significant  skill  recovery.  After  this  recovery, 
there  was  no  evidence  that  Last  Day  30  psychomotor  trial  times 
differed  significantly  from  the  Last  Day  1  times. 

IVIS  Declarative  knowledge  retention.  Declarative 
knowledge  test  scores  showed  a  significant  1.86  point  (29%) 
decrease  across  the  30  day  retention  interval,  followed  by  a 
significant  1.52  point  (24%)  improvement  over  Day  30  trials. 

The  comparison  of  Last  Day  1  and  Last  Day  30  test  scores  failed 
to  provide  evidence  of  lasting  decay  in  soldiers  declarative 
knowledge  of  IVIS  overlay  and  report  tasks  after  the  30  day 
retention  interval . 
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The  remainder  of  the  results  section  presents  a  detailed 
review  of  the  findings  for  each  of  the  six  research  issues 
identified  previously  in  Table  1.  The  six  issues  address  Day  1 
skill  acquisition,  Day  30  skill  retention,  IVIS  subtask 
performance,  IVIS  task  step  performance,  psychomotor  skill,  and 
declarative  knowledge  skill. 

Issue  1.  Digital  Skill  Acquisition 

The  issue  of  IVIS  skill  retention  could  not  be  addressed 
without  first  determining  whether  skills  had  been  acquired 
during  the  Day  1  train-up  and  evaluation  session.  Not  all 
participants  learned  how  to  perform  IVIS  tasks  on  Day  1.  It 
would  confound  estimates  of  skills  retention  to  measure  their 
performance  again  on  Day  30  and  attribute  low  performance  to 
skill  decay,  rather  than  the  failure  to  acquire  the  skills  in 
the  first  place.  Therefore,  those  who  failed  to  acquire  skills 
on  Day  1  were  identified  and  eliminated  from  the  sample  when 
estimating  skill  decay  over  the  30  day  retention  interval. 

Criteria  for  IVIS  skill  acquisition.  Summary  tables  of 
overlay  and  report  trial  success  rate  and  performance  times  for 
all  soldiers  and  all  trials  are  provided  as  Appendix  K  and  L, 
respectively.  Table  4  presents  the  number  of  Day  1  training 
session  overlay  and  report  tasks  that  soldiers  performed 
correctly.  From  the  distribution  of  scores  it  was  decided  to 
select  those  participants  who  successfully  completed  three  or 
more  of  the  four  overlay  tasks  as  having  successfully  acquired 
the  overlay  skills,  so  that  the  criteria  for  IVIS  skills 
acquisition  was  a  75%  or  better  success  rate  on  Day  1  tasks. 
Using  this  criteria  22  of  the  participants  (78.5%)  were  judged 
to  have  acquired  IVIS  overlay  skill  on  Day  1,  and  were  referred 
to  as  the  Overlay  Skilled  group.  In  responding  to  IVIS  System 
Procedures  Review  1  soldiers  rated  the  training  provided  for 
overlays  as  "Very  Adequate"  (89%)  or  Fairly  Adequate  (11%),  and 
the  training  provided  for  reports  as  "Very  Adequate"  (93%)  or 
"Fairly  Adequate"  (7%) . 

Table  4  also  presents  the  number  of  Day  1  training  session 
report  tasks  performed  correctly.  From  the  distribution  of 
scores  it  was  decided  to  select  those  who  successfully  completed 
two  or  more  of  the  three  report  tasks  as  having  successfully 
acquired  the  reports  skill,  so  that  the  criteria  for  acquisition 
was  a  66%  success  rate  or  better.  Using  this  criteria  for 
report  skill  acquisition,  23  participants  (82%)  were  judged  to 
have  acquired  IVIS  report  skills  on  Day  1,  and  were  referred  to 
as  the  Report  Skilled  group.  One  soldier  was  transferred  after 
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the  Day  1  evaluations,  reducing  the  Day  30  sample  size  to  27, 
with  21  soldiers  remaining  from  the  Overlay  Skilled  group,  and 
22  remaining  from  the  Report  Skilled  group. 

Table  4 

Number  of  Soldiers  Successfully  Completing  Day  1  Trials 


Successful 

Trials 

Overlay 

Report 

Number 

(percent) 

Number  (percent) 

0 

4 

(14) 

2  (  7) 

1 

0 

(  0) 

3  (11) 

2 

2 

(  7) 

9  (32) 

3 

13 

(46) 

14  (50) 

4 

9 

(33) 

Total 

28 

(100) 

28  (100) 

Note.  There  were  four  overlay  and  three  report  Day  1  trials. 


The  criteria  for  IVIS  overlay  skill  acquisition  (75%) 
differed  from  the  criteria  for  reports  (66%)  due  to  the 
different  number  of  tasks  performed,  so  that  a  direct  comparison 
of  acquisition  rates  was  not  possible.  The  proportion  of 
overlay  tasks  successfully  completed  for  the  Skill  Acquired,  and 
Not  Acquired  groups  is  provided  as  Figure  5.  The  figure 
suggests  that  soldiers  that  did  not  acquire  skills  have  low 
success  rates  in  both  the  Day  1  and  Day  30  evaluations. 
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Figure  5.  Soldiers  successfully  completing  overlay  tasks. 
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The  proportion  of  report  tasks  successfully  completed  for 
the  Skill  Acquired,  and  Not  Acquired  groups  is  provided  as 
Figure  6.  The  figure  suggests  that  soldiers  that  did  not 
acquire  report  skills  have  low  success  rates  in  both  the  Day  1 
and  Day  30  evaluations  compared  to  the  skilled  group. 
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Day  1  Trials  Day  30  Trials 

Figure  6.  Soldiers  successfully  completing  report  tasks. 

The  performance  time  and  success  rate  data  for  IVIS  overlay 
and  report  trials  for  the  Skilled  Groups  are  provided  as 
Appendix  M  and  N  respectively.  It  should  be  noted  that  the  task 
performance  times  include  times  for  successful  tasks  only.  For 
some  tasks  there  were  very  few  successful  trials,  resulting  in 
very  little  time  data.  As  an  example,  only  six  participants 
successfully  completed  Overlay  Trial  8,  which  immediately 
follows  the  30  day  delay  period.  An  outlier  analysis  was 
conducted  to  identify  extreme  performance  time  values  falling 
more  than  three  standard  deviations  beyond  the  mean  for  a 
particular  trial.  The  total  data  set  included  385  time  values 
(7  trials  per  day,  with  28  Day  1  and  27  Day  30  soldiers) .  From 
the  total  of  385  time  values,  five  outlier  values  (one  overlay, 
four  report)  were  identified  using  the  three  standard  deviation 
criteria.  For  each  outlier  value  a  "Winsorizing"  procedure 
suggested  by  Marascuilo  and  McSweeney  (1977)  was  applied  where 
the  largest  (outlier)  time  value  was  replaced  with  the  next 
largest  value,  and  the  smallest  time  value  was  also  replaced 
with  the  next  smallest  value. 
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One  IVIS  instructor  served  as  a  subject  matter  expert  (SME) 
for  the  experiment,  and  provided  task  performance  times  that 
appear  as  a  benchmark  reference  in  overlay  and  report 
performance  time  figures.  The  SME  performed  four  overlay  and 
three  report  using  experimental  materials,  and  following 
experimental  procedures.  The  SME  was  instructed  to  perform  the 
tasks  at  a  pace  that  would  be  appropriate  for  an  operational 
environment.  The  mean  times  for  successful  task  completion  were 
150  seconds  for  overlays,  and  78  seconds  for  reports. 

Perceived  training  adequacy.  Questions  presented  in  IVIS 
System  Procedures  Review  1  asked  soldiers  to  rate  the  quality  of 
the  experiment  training  they  had  received  on  Day  1 .  Soldiers 
rated  the  training  for  overlay  tasks  as  "Very  Adequate"  (89%), 
and  "Fairly  Adequate"  (11%),  and  rated  report  task  training  as 
"Very  Adequate"  (93%)  and  "Fairly  Adequate"  (7%) .  These 
findings  provide  some  evidence  that  soldiers  perceived  the  IVIS 
training  as  having  adequately  prepared  them  to  perform  the 
experimental  tasks . 

Issue  2.  Digital  Skill  Retention 

IVIS  Overlay  skill  retention;  successful  task  performance. 
Figure  7  presents  the  proportion  of  Overlay  Skilled  soldiers 
successfully  completing  Day  1  and  Day  30  overlay  tasks. 

Appendix  0  provides  the  summary  results  of  statistical 
comparisons  of  overlay  trial  success  rates  using  the  McNemar 
test  for  correlated  proportions  with  exact  (one-tailed)  binomial 
estimate  (McNemar,  1975) . 


Figure  7 .  Proportion  of  Overlay  Skilled  soldiers  successfully 
completing  overlays  for  comparison  trials. 
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Considering  first  the  issue  of  learning  during  Day  1 
trials,  overlay  task  performance  showed  only  slight  improvement 
across  tasks.  The  comparison  of  the  proportion  of  soldiers 
successful  on  First  Day  1  (P  =  .82)  and  Last  Day  1  trials 
(P  =  0.86)  yields  a  difference  of  only  .04,  which  was  not 
significant.  With  a  =  .05,  n  =  22,  power  =  .09,  so  that  there 
is  only  a  .09  probability  that  the  test  would  find  a  significant 
difference  of  this  size  if  it  existed. 

Examining  skill  retention  after  30  days  and  prior  to 
practice  on  all  seven  Day  30  trials,  the  comparison  of  Day  1  and 
Day  30  overlay  task  success  rates  provides  evidence  of 
significant  skill  decay  over  time.  The  decrease  in  task  success 
from  Last  Day  1  (P  =  .86)  to  First  Day  30  (P  =  .29)  yielded  a 
difference  of  .57  which  was  significant,  n  =  21,  £  =  .001, 
power  =  .98.  This  finding  suggests  that  participants 
experienced  significant  overlay  skill  decay  over  the  30  day 
retention  period. 

Considering  skill  retention  after  30  days,  and  after 
practice  with  the  seven  Day  30  tasks,  the  0.38  success  rate 
decrease  from  Last  Day  1  (P  =  .86)  to  Last  Day  30  (P  =  .48)  is 
also  significant  n  =  21,  p  =  .029,  power  =  .74.  This  comparison 
provides  evidence  that  skills  were  not  recovered  with  practice 
on  Day  30  tasks  to  the  level  of  Day  1  performance. 

As  a  final  estimate  of  retention,  the  overlay  skill 
acquisition  criteria  of  three  or  more  successful  trials  was 
applied  to  Day  30  task  performance  to  identify  those  soldiers 
who  remained  Overlay  Skilled.  An  advantage  associated  with 
comparisons  based  on  this  criteria  was  that  it  incorporated 
performance  data  from  all  eight  overlay  trials,  as  opposed  to 
single  trial  retention  comparisons.  On  Day  30  overlay  trials  10 
out  of  21  Overlay  Skilled  soldiers  successfully  completed  three 
or  more  tasks  and  again  met  the  overlay  skill  acquisition 
criteria,  while  11  Overlay  Skilled  soldiers  failed  to  achieve 
this  criteria  for  proficiency.  Using  the  McNemar  test  with 
exact  significance  (1-tailed)  binomial  estimate  (McNemar,  1975) , 
the  .52  decline  in  the  proportion  of  soldiers  meeting  the 
Overlay  Skilled  criteria  from  the  Day  1  (P  =  1.00)  to  Day  30 
(P  =  .48)  was  significant,  n  =  21,  p  =  0.000,  power  =  1.00,  and 
provided  further  evidence  of  decay  in  overlay  skill  over  the  30 
day  retention  period. 

Examining  overlay  skill  recovery  with  practice  during  the 
Day  30  trials,  the  change  in  success  rate  from  the  First  Day  30 
(P  =  .29)  to  the  Last  Day  30  (P  =  .48)  trials  yields  a 
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difference  of  .19,  which  was  not  significant.  With  a  =  .05,  n  = 
21,  power  =  .53.  No  statistically  significant  differences  were 
found  between  the  compared  trials  success  rates  to  provide 
information  about  skill  recovery  on  Day  30  as  resulting  from 
practice  on  the  tasks. 

IVIS  Report  skill  retention:  successful  task  performance. 
There  are  important  differences  between  the  sequencing  of 
overlay  and  report  tasks  which  must  be  considered  in  reviewing 
these  results.  The  first  and  last  trials  for  both  Day  1  and  Day 
30  are  overlay  trials,  so  the  participant  receives  one  overlay 
task  trial  after  the  last  report  task  on  Day  1,  and  one  overlay 
task  trial  before  performing  the  first  report  task  on  Day  30. 

The  intervention  of  two  overlay  tasks  between  the  Last  Day  1  and 
First  Day  30  report  comparisons  provides  extra  practice  which 
could  result  in  a  biased  estimate  of  report  skill  retention  over 
the  30  day  delay.  Here  the  intervening  overlay  tasks  might 
provide  general  IVIS  practice  reminding  the  soldier  how  to  do 
reports  (a  positive  bias),  or  they  might  produce  specific 
procedural  confusion  interfering  with  report  performance  (a 
negative  bias) . 

The  proportion  of  Report  Skilled  soldiers  successfully 
completing  First  and  Last  Day  1  and  Day  30  trials  is  presented 
in  Figure  8.  Appendix  P  provides  the  summary  results  of 
statistical  comparisons  of  trial  success  rates  using  the  McNemar 
test  for  correlated  proportions  with  exact  significance 
(1-tailed)  binomial  estimate  (McNemar,  1975) . 

Looking  first  at  Day  1  learning,  there  was  a  slight 
increase  of  .08  in  success  rate  from  First  Day  1  (P  =  .83)  to 
Last  Day  1  (P  =  .91)  trials  which  was  not  significant.  With 
a  =  .05,  n  =  22,  power  =  .09.  For  this  small  difference  power 
is  extremely  low,  and  no  statically  significant  differences 
associated  with  learning  were  found  between  the  compared  trials 
success  rates. 

Examining  the  retention  of  report  skill  over  the  30  day 
retention  interval,  and  without  practice  on  the  seven  Day  30 
tasks,  the  .14  decline  in  success  from  Last  Day  1  (P  =  .91)  to 
First  Day  30  (P  =  .77)  trials  was  not  significant.  With 
a  =  .05  ,  n  =  21,  power  =  .18,  so  that  there  is  only  a  .18 
probability  that  the  test  would  find  an  effect  of  this  size  if 
it  exists.  No  statically  significant  differences  were  found 
between  the  proportion  of  soldiers  successful  on  the  compared 
trials  to  provide  information  about  skill  decay  over  the  30  day 
interval . 
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Considering  report  skill  retention  after  soldiers  completed 
all  seven  Day  30  trials,  the  decrease  in  successful  performance 
from  Last  Day  1  (P  =  .91)  to  Last  Day  30  (P  =  .77),  yielded  a 
.14  decrease  in  success  which  was  not  significant.  With  a  = 

.05,  n  =  22,  power  =  .18,  and  no  statically  significant 
differences  were  found  regarding  skill  retention. 

Estimating  skill  retention  using  the  report  skill 
acquisition  criteria  of  two  or  more  successful  trials,  the 
decline  in  the  proportion  of  Report  Skilled  soldiers  from  Day  1 
(P  =  1.00)  to  Day  30  (P  =  .77)'  yielded  a  difference  of  .23. 

This  difference  was  significant,  n  =  22,  p  =  .002,  power  =  1.00. 
Using  the  data  from  all  three  Day  30  report  trials,  the  skill 
acquisition  performance  criteria  provided  evidence  of  a 
significant  .23  decrease  in  report  skill  over  the  30  day 
retention  period. 

Considering  report  skill  recovery  with  practice  during  Day 
30  trials,  the  success  rate  for  reports  did  not  change  from  the 
First  Day  30  (P  =  .77)  to  the  Last  Day  30  (P  =  .77)  trials. 

With  a  =  .05,  n  =  22,  power  =  .05,  and  no  significant 
differences  were  found. 


First _ Last 

Day  1  Trials 


First _ Last 
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Figure  8.  Proportion  of  Report  Skilled  soldiers  successfully 
completing  reports  for  comparison  trials. 
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I VIS  Overlay  skill  retention:  performance  time.  The  IVIS 
Overlay  task  performance  times  for  successful  trials  were 
compared  to  provide  estimates  of  overlay  skill  learning, 
retention,  and  skill  recovery.  A  summary  of  overlay  task 
performance  times  for  the  Overlay  Skilled  group  is  provided  as 
Figure  9 .  Appendix  Q  provides  the  summary  results  of  paired 
t-tests  (2-tailed)  comparing  overlay  trial  times.  It  should  be 
noted  that  the  mean  trial  performance  time  and  sample  size 
differs  across  trials  based  on  the  number  of  soldiers  having 
valid  data  for  both  trials  in  a  specific  comparison.  Only 
successful  trial  performance  times  are  included  in  calculations, 
so  that  comparison  sample  size  ranges  from  a  low  of  five,  to  a 
high  of  15  soldiers. 

600  t - — - — - 1 


Day  1  Trials  Day  30  Trials 


Figure  9 .  Overlay  task  performance  times  for  comparison  trials . 
Note.  The  dashed  line  represents  the  150  second  mean  time 
required  by  the  IVIS  SME  to  successfully  complete  overlay  tasks. 

Examining  first  the  issue  of  learning  during  Day  1  trials, 
the  time  (in  seconds)  required  to  successfully  complete  Day  1 
trials  showed  a  significant  decline  from  First  Day  1  (M  =  214, 

SD  =  62)  to  Last  Day  1  trials  (M  =  179,  SD  =  36)  ,  t(14)  =  2.994, 
p  <  .01,  power  =  .80  (paired  t-test,  2-tailed).  This  reduction 
in  task  performance  time  provided  evidence  that  significant 
overlay  skill  learning  was  taking  place  during  the  Day  1  trials, 
beyond  what  had  been  achieved  in  the  Day  1  training. 
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Investigating  skill  retention  immediately  following  the  30 
day  retention  interval,  the  comparison  of  Day  1  and  Day  30 
overlay  task  performance  times  provided  evidence  of  significant 
skill  decay.  The  321  second  increase  in  performance  time  from 
Last  Day  1  (M  =  197,  SD  =  48)  to  First  Day  30  (M  =  518,  SD  = 

222)  was  significant,  t(4)  =  3.861,  p  <  .05,  power  =  .82, 
suggesting  that  participants  were  less  proficient  in  performing 
I VIS  overlay  tasks  immediately  after  the  30  day  retention 
period. 

Considering  whether  skill  decay  remains  on  Day  30  after 
soldiers  had  practice  with  the  seven  Day  30  trial  tasks,  the  18 
second  decrease  in  performance  time  from  Last  Day  1  (M  =  178,  SD 
=  51)  to  Last  Day  30  (M  =  160,  SD  =  31)  was  not  significant 

t (6)  =  1.723.  With  a  =  .05,  n  =  7,  and  an  effect  size  of  18 

seconds,  power  =  .31.  For  a  time  difference  this  small  power  is 
extremely  low,  and  no  statistically  significant  differences  were 
found  between  the  compared  trial  times . 

Investigating  whether  skill  recovery  occurred  with  practice 
on  Day  30  trials,  the  315  second  reduction  in  successful  task 
performance  times  from  the  First  Day  30  (M  =  491,  SD  =  218)  to 
the  Last  Day  30  (M  =  176,  SD  =  33)  trials  was  significant, 

t (4)  =  3.688,  p  <  .05,  power  =  .78,  and  provided  evidence  of 

overlay  skill  recovery  with  practice  during  the  Day  30  trials. 

Performance  time  appears  to  be  an  essential  criteria  for 
estimating  IVIS  task  performance.  For  the  present  study  the 
overlay  task  success  rate  did  not  improve  across  Day  1  trials. 
However,  the  significant  reduction  in  task  performance  time  did 
provide  evidence  of  skill  recovery  with  unaided  practice. 

IVIS  Report  skill  retention;  performance  time.  A  summary 
of  successful  report  task  performance  times  for  the  Report 
Skilled  group  is  provided  as  Figure  10,  while  Appendix  R 
provides  the  summary  results  of  paired  t-tests  comparing 
successful  report  trial  times.  Examining  first  the  issue  of 
additional  skill  learning  during  Day  1  trials,  the  time  required 
to  successfully  complete  Day  1  trials  showed  a  47  second  decline 
from  First  Day  1  (M  =  148,  SD  =  39)  to  Last  Day  1  (M  =  101,  SD  = 
18)  trials  which  was  significant,  t(16)  =  6.208,  p  <  .001,  power 
=  1.00  (paired  t-test,  2-tailed).  This  reduction  in  task 
performance  time  provides  some  evidence  that  report  skill 
learning  was  taking  place  during  the  Day  1  trials,  beyond  what 
had  been  achieved  in  Day  1  training. 
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Considering  skill  retention  without  practice  on  Day  30,  the 
comparison  of  Day  1  and  Day  30  report  task  performance  times 
provided  evidence  of  significant  skill  decay  over  time.  The  52 
second  increase  in  performance  time  from  Last  Day  1  (M  =  102 ,  SD 
=  25)  to  First  Day  30  (M  =  154,  SD  =  45)  was  significant,  t(15) 

=  5.383,  p  <  .001,  power  =  1.00.  This  increase  in  performance 
time  suggests  that  participants  were  less  proficient  in 
performing  IVIS  report  tasks  immediately  after  the  30  day 
retention  period,  and  prior  to  practice  with  the  seven  Day  30 
trials . 
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Figure  10.  Report  task  performance  times  for  comparison  trials. 
Note.  The  dashed  line  represents  the  78  second  mean  time 
required  by  the  IVIS  SME  to  successfully  complete  report  tasks. 

Examining  whether  report  skill  decay  remained  after 
practice  with  Day  30  trials,  performance  time  increased  15 
seconds  from  Last  Day  1  (M  =  104,  SD  =  25)  to  Last  Day  30 
(M  =  119,  SD  =  44)  which  was  not  significant,  t(15)  =  1.327, 
power  =  .24. 

Investigating  whether  there  was  skill  recovery  with  practice 
during  the  Day  30  trials,  the  72  second  performance  time 
decrease  from  the  First  Day  30  trials  (M  =  187,  SD  =  124)  to  the 
Last  Day  30  trials  (M  =  115,  SD  =  46)  was  not  significant.  With 
a  =  .05,  n  =  15,  and  an  effect  size  of  71  seconds,  power  =  .52, 
and  no  statistically  significant  differences  between  the 
compared  trials  were  found.  While  the  comparison  of  trials 
yigl(3.s  &  large  time  difference,  the  standard  deviation  of  First 
Day  30  trials  is  also  quite  large,  which  lowers  power  and 
impacts  the  significance  test. 
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Predicting  skill  retention  with  the  User's  Decision  Aid. 

The  User's  Decision  Aid  skill  retention  rating  scale  developed 
by  Rose,  Czarnolewski,  Gragg,  Austin,  Ford,  Doyle,  and  Hagman 
(1985)  was  employed  in  the  present  experiment  to  provide  a  skill 
retention  estimate  that  represents  the  expected  proportion  of 
soldiers  in  a  unit  able  to  perform  a  task  correctly  after  a 
period  of  one  month  without  training.  The  author  and  primary 
instructor  rated  the  overlay  and  report  tasks  presented  in  the 
experiment  using  the  ten  item  task  characteristic  rating  scale. 
Individual  ratings  for  the  ten  scale  items  are  provided  as 
Appendix  S.  The  resulting  predicted  retention  estimates  were 
compared  to  the  observed  Day  30  overlay  and  report  task  success 
rates.  Specific  User's  Decision  Aid  retention  estimates  are 
based  on  the  assumption  that  100%  of  soldiers  were  initially 
trained  to  proficiency.  The  IVIS  skills  acquisition  criteria  of 
three  or  more  successful  overlay  tasks,  and  two  or  more 
successful  report  tasks  was  used  to  establish  groups  of  soldiers 
with  100%  Day  1  skills  proficiency,  and  was  also  employed  as  the 
equivalent  Day  30  skill  retention  criteria. 

A  binomial  probability  test  for  comparing  observed  values 
against  the  predicted  retention  proportion  was  used  for  the 
evaluation.  A  two-tailed  test  was  conducted  as  ratings  might 
over  predict,  or  under  predict  observed  values.  Using  Cohen's 
(1988)  convention  for  values  associated  with  a  large  effect  size 
(.75  vs.  .50),  with  a  =  .05  (2-tailed),  n  =  21,  power  for  the 
comparisons  would  be  . 67 . 

The  skill  retention  rating  method  predicted  that  .67  of 
the  Overlay  Skilled  group  would  achieve  the  proficiency  criteria 
again  on  Day  30.  In  Day  30  hands-on  testing  10  soldiers  (48%) 
successfully  met  the  criteria  by  completing  three  or  more 
overlay  trials.  A  binomial  probability  test  comparing  the 
observed  against  the  predicted  retention  proportion  was  not 
significant.  The  95%  confidence  interval  for  the  population 
proportion  shows  that  the  values  of  the  proportion  supported  by 
the  data  lie  between  .29  and  .68.  For  the  given  effect  size 
(population  proportion  =  0.48,  tested  against  a  constant  of 
.67),  n  =  21,  and  a  =  .05,  (2-tailed),  power  was  .40.  Here,  40% 
of  studies  would  be  expected  to  yield  a  significant  effect, 
rejecting  the  null  hypothesis  that  the  population  proportion 
was  . 67 . 

For  the  report  task  22  soldiers  remained  in  the  Day  30 
Report  Skilled  sample.  The  skill  retention  rating  method 
predicted  that  .92  of  the  Report  Skilled  group  would  achieve  the 
Skill  Acquired  criteria  again  on  Day  30.  In  Day  30  hands-on 
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testing  17  soldiers  (77%)  successfully  met  the  criteria  by 
completing  two  or  more  report  trials.  A  binomial  probability 
test  comparing  the  observed  against  the  predicted  retention 
proportion  was  significant,  n  =  22,  p  =  .01,  power  =  .40.  The 
95%  confidence  interval  for  the  population  proportion  shows  that 
the  values  of  the  proportion  supported  by  the  data  lie  between 
.56  and  .90.  Here,  40%  of  studies  would  be  expected  to  yield  a 
significant  effect,  rejecting  the  null  hypothesis  that  the 
population  proportion  is  .92. 

Issue  3 :  Digital  Skill  Subtask  Assessment 

Measuring  performance  at  the  subtask  level  provides  a 
useful  way  of  summarizing  and  organizing  performance  information 
that  can  help  training  developers  to  focus  their  limited 
resources  on  specific  problem  areas.  In  an  effort  to  identify 
where  performance  problems  occur,  and  how  training  might  be 
modified  to  address  these  problems,  the  IVIS  overlay  and  report 
tasks  were  divided  into  subtasks.  For  this  research,  subtasks 
were  defined  as  clusters  of  separate  actions  which  must  be 
performed  as  a  step-by-step  sequence.  Four  overlay  subtasks 
were  identified  from  a  review  of  the  23  action  steps  required 
for  task  performance,  and  incorporated  into  the  video  data 
reduction  sheets  used  to  capture  performance  data.  The  subtasks 
paralleled  the  instructional  subdivisions  in  the  M1A2  NETT 
lesson  plan. 

The  four  IVIS  overlay  subtasks  identified  for  this  research 

were: 

1.  Navigate:  Navigate  through  mode  and  menu  options  to 
arrive  at  the  IVIS  graphics  menu. 

2.  Graphic:  Use  menu  options  and  cursor  controller  to 
place  a  two -point  graphic  on  the  map  display. 

3.  TRP:  Use  menu  options  and  keypad  to  place  a  Target 
Reference  Point  on  the  mapi. 

4.  Send:  Using  menu  options  send  the  overlay  information. 

Five  report  subtasks  were  also  identified  from  a  review  of 
the  19  action  steps  required  for  task  performance,  and 
incorporated  into  the  video  data  reduction  sheets  used  to 
capture  performance  data.  Report  subtasks  identified  for  this 
research  were: 

1.  Navigate:  Navigate  through  mode  and  menu  options  to 
arrive  at  the  IVIS  Spot  Report  menu. 
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2.  Target  Grid:  Use  thumb  cursor  to  place  a  target  icon 
on  the  map  display. 

3.  Target  Type:  Use  menu  options  to  select  target  type, 
subtype,  and  number  of  targets. 

4.  Activity:  Use  menu  options  to  enter  Friendly  Action 
and  Enemy  Activity  description. 

5.  Send:  Using  menu  options  send  the  report  information. 

Overlay  subtask  performance.  Using  the  video  data 
reduction  sheets,  task  performance  errors  were  assigned  codes 
identifying  the  subtask  to  which  they  belonged.  The  frequency 
of  overlay  subtask  errors  occurring  in  the  first  and  last  trials 
on  Day  1  and  Day  30  for  the  full  sample  of  28  soldiers  is 
presented  in  Figure  11.  From  this  figure  it  appears  that  the 
majority  of  overlay  task  performance  errors  occurred  in  the  TRP 
and  Send  subtasks,  with  an  increase  in  errors,  after  the  30  day 
delay.  These  results  might  be  used  to  focus  limited  training 
resources  on  the  TRP  and  Send  subtask  problem  areas  that  show 
high  payoff  potential  in  terms  of  error  reduction. 


First _ Last  First _ Last. 

Day  1  Trials  Day  30  Trials 


Figure  11.  Frequency  of  overlay  subtask  errors  across  trials. 

Report  subtask  performance.  The  frequency  of  report 
subtask  errors  occurring  in  the  first  and  last  trials  on  Day  1 
and  Day  30  for  the  Report  Skilled  group  is  presented  in  Figure 
12.  Day  1  errors  primarily  occurred  on  the  Target  Grid,  Target 
Type,  and  Enemy  Action  subtasks,  with  no  errors  occurring  for 
the  Navigation  subtask.  For  Day  30  trials  the  error  pattern 
changes  somewhat,  as  soldiers  made  errors  with  Grid  coordinates 
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while  Target  Type,  Enemy  Action,  and  report  Send  subtasks  showed 
increased  errors.  These  findings  would  suggest  that  report  task 
errors  were  not  very  frequent  and  were  distributed  across  all 
subtasks  except  Navigation. 
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Figure  12.  Frequency  of  report  subtask  errors  across  trials. 

Subtask  time  assessment.  The  problem  of  obtaining 
performance  time  data  for  unsuccessful  trials  can  be  partly 
resolved  by  subdividing  tasks  into  successful  and  unsuccessful 
subtasks,  so  that  time  data  can  be  obtained  from  successful 
subtasks  occurring  within  unsuccessful  trials.  For  the  present 
research  IVIS  overlay  subtask  performance  times  were  recorded 
during  video  data  reduction  using  the  video  data  reduction 
sheets  (Appendix  B) .  Results  of  the  First  Day  30  overlay  trial 
for  the  Overlay  Skilled  group  provides  an  example  of  the 
potential  utility  of  subtask  level  task  time  assessment.  Only 
six  participants  successfully  completed  this  task,  providing 
only  six  successful  task  performance  times.  By  separating  the 
overlay  task  into  subtasks,  successful  subtasks  providing 
additional  useful  time  data  were  identified.  Compared  to  only 
six  successful  First  Day  30  overlay  task  times,  there  were  First 
Day  30  times  available  for  19  successful  Navigation  subtasks,  17 
Graphic  subtasks,  16  TRP  subtasks,  and  11  Send  subtasks. 

Figure  13  presents  a  comparison  of  overlay  subtask 
completion  time  across  trials  for  the  full  population  of  28 
soldiers.  This  figure  suggests  that  while  there  was  a  general 
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increase  in  time  required  to  perform  subtasks  immediately 
following  the  30  day  retention  period,  the  TRP  subtask  in 
particular  appears  to  have  contributed  to  the  increased  First 
Trial  Day  30  time  requirement.  Using  performance  time  as  the 
criteria  it  appears  that  the  TRP  subtask  represents  the  area 
where  the  greatest  payoff  might  be  gained  during  efforts  to 
focus  training  to  reduce  task  performance  time. 
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Figure  13.  Overlay  subtask  completion  times  across  trials. 

Issue  4.  Digital  Skill  Task  Step  Assessment 

Specific  subtask  step  errors  were  scored  during  data 
analysis  as  a  means  of  identifying  where  performance  problems 
occurred,  and  the  nature  of  the  problem.  A  summary  of  frequent 
Day  30  IVIS  overlay  and  report  task  step  errors  for  each  subtask 
is  presented  as  Appendix  T.  For  each  step  error  a  description 
of  the  error  and  possible  corrective  actions  are  provided.  A 
complete  list  of  overlay  and  report  task  step  errors  for  the 
full  sample  of  28  soldiers  by  frequency  and  type  appears  in 
Appendix  U. 

Identifying  sources  of  performance  errors.  Error 
descriptions  in  Appendix  T  might  be  useful  to  training 
developers  in  suggesting  specific  task  steps  to  focus  training 
revision  efforts  on,  whether  through  changes  in  classroom 
training  or  equipment  design.  As  example,  for  both  the  Navigate 
and  Send  subtasks,  participants  would  enter  a  software  display 
and  fail  to  recognize  that  the  system  required  them  to  select 


34 


one  IVIS  overlay  from  a  list  prior  to  moving  on  to  the  next  task 
step.  No  cue  was  provided  to  tell  the  operator  to  "Choose  an 
Overlay."  With  regard  to  the  placement  of  Graphics,  some, 
participants  failed  to  remember  the  basic  procedure  for  finding 
a  location  on  a  map.  With  regard  to  the  TRP  subtask,  errors 
here  reflected  a  failure  to  remember  material  provided  in  the 
Day  1  lecture,  and  the  instructions  written  on  the  task  card, 
where  participants  were  told  that  they  must  use  the  IVIS  keypad 
to  enter  precise  TRP  coordinates. 

Some  problems  appeared  to  reflect  a  logical  (but  false) 
assessment  of  the  literal  meaning  of  system  cues.  In  the 
overlay  task,  soldiers  often  selected  "CONTINUE"  when  attempting 
to  begin  the  second  graphic,  instead  of  the  correct  "RETURN" 
menu  option.  In  the  report  task  soldiers  forgot  that  the  "SEND" 
option  doesn't  actually  send  the  report.  Having  identified 
these  specific  misunderstandings,  additional  training  emphasis 
could  be  selectively  focused  on  reinforcing  the  correct  meaning 
of  these  menu  options,  emphasizing  for  example,  that  "the  SEND 
option  does  not  actually  send  the  overlay  or  report." 

While  investigating  report  task  performance  errors  it  was 
noted  that  there  was  no  procedure  in  the  IVIS  lesson  plan  for 
clearing  target  data  errors  from  a  Spot  Report.  This  problem 
was  compounded  by  the  software  design,  which  retains  target  data 
in  the  Spot  Report  even  if  the  soldier  tries  to  clear  all 
entries  by  quitting  without  sending  the  report.  Having, 
identified  the  problem,  the  necessary  procedure  to  eliminate  bad 
data  could  be  included  in  a  revision  of  the  lesson  plan. 

Soldiers  (predictably)  showed  tendencies  toward  caution  in 
performing  IVIS  tasks.  In  the  absence  of  obvious  "Mail  Sent" 
feedback  from  the  system,  soldiers  would  send  duplicate  messages 
over  multiple  nets.  These  performance  errors  have  little  impact 
in  a  training  environment,  but  could  overload  a  digital 
communications  network  during  combat  operations  with  blank 
reports,  redundant  information,  and  inappropriate  message 
traffic.  By  scoring  and  quantifying  this  behavior,  the  training 
manager  can  identify  where  soldiers'  good -intentions  lead  to  bad 
habits,  and  the  need  for  warnings  against  this  behavior  in  the 
lesson  plan.  Even  a  simple  list  of  frequently  made  errors  could 
provide  a  trainer  with  a  useful  set  of  warnings  to  discuss  with 
soldiers  receiving  training. 

Feedback  to  soldiers  during  training .  The  identification 
of  IVIS  subtask  and  step  errors  suggests  information  that  might 
be  provided  as  feedback  to  the  soldier  during  training.  The 


35 


XVIS  equipment  presents  a  prompt  window  asking  the  operator  to 
save  his  work  if  he  attempts  to  leave  the  graphics  mode  without 
first  selecting  the  Save  option.  IVIS  detects  that  new  graphics 
have  been  created  but  not  saved,  and  prompts  the  operator  to  the 
appropriate  action.  This  same  functionality  might  be  applied  to 
other  task  steps  that  were  found  to  have  high  error  rates.  As 
example:  The  most  frequently  encountered  errors  for  the  First 
Day  30  overlay  trials  involved  participants  attempting  to  create 
graphics ,  or  send  a  completed  overlay,  without  first  identifying 
which  overlay  was  to  be  created  or  sent.  Using  the  current 
functionality  of  the  IVIS  "SAVE"  prompt,  the  system  might 
provide  the  prompt  "Select  Overlay"  to  remind  the  soldier  that 
this  task  step  must  be  completed. 

Issue  5.  IVIS  Psychomotor  Skill  Assessment 

In  examining  the  contribution  of  psychomotor  skill  to  IVIS 
task  performance  it  was  necessary  to  investigate  whether  this 
skill  showed  consistency  across  trials,  suggesting  that  it  might 
represent  an  individual  difference  characteristic.  The 
relationship  between  psychomotor  skills  and  IVIS  overlay  and 
report  task  performance  was  also  investigated  to  identify 
whether  these  skills  might  make  a  significant  contribution  to 
task  performance.  Finally,  psychomotor  skills  were  investigated 
to  identify  whether  they  showed  a  pattern  of  retention  over  time 
similar  to  that  displayed  for  IVIS  task  performance.  If 
psychomotor  skills  were  related  to  task  performance,  and  showed 
decay  over  the  30  day  retention  interval,  then  these  skills 
might  represent  a  high  priority  training  requirement,  and 
justify  the  expense  of  the  high  fidelity  CCHA  IVIS  interface 
training  equipment . 

The  full  sample  of  28  soldiers  was  used  in  the  psychomotor 
analyses  to  provide  the  broadest  possible  range  of  performance 
values,  including  those  for  soldiers  who  failed  to  meet  the  IVIS 
Skill  Acquired  performance  criteria  on  Day  1.  Each  of  the  28 
soldiers  performed  four  psychomotor  tasks  for  a  total  of  110 
trials  (one  soldier  was  transferred  and  failed  to  complete  the 
two  Day  30  trials) .  Task  time  was  used  as  the  performance 
criteria  for  the  analyses,  but  task  success  was  not,  as  there 
were  only  five  examples  of  partial  task  failure  (failure  to 
highlight  a  number)  across  the  110  trials. 

Psychomotor  skill  consistency  across  trials.  A  correlation 
matrix  was  calculated  to  investigate  whether  psychomotor  skill 
proficiency  was  related  to  overlay  and  report  Day  30  task 
performance  (see  Appendix  V) .  From  this  matrix  it  was  observed 
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that  First  Day  1  psychomotor  task  performance  times  were 
significantly  related  to  Last  Day  1  times  (r  =  .588,  p  <  .001, 
First  Day  30  times  (r  =  .642,  p  <  .001),  and  Last  Day  30  times 
(r  =  .450,  p  <  .001).  These  findings  provide  some  evidence  that 
psychomotor  task  performance  times  showed  consistency  across 
trials,  and  might  represent  an  individual  characteristic. 

Psychomotor  skill  relationship  to  IVIS  task  success.  The 
correlation  matrix  presented  in  Appendix  N  provides  a  comparison 
between  the  four  psychomotor  trials  and  the  total  number  of 
successful  Day  1  and  Day  30  overlay  and  report  trials.  Results 
of  the  16  comparisons  identified  only  a  single  significant 
relationship  between  Last  Day  30  psychomotor  trial  time  and  the 
sum  of  successful  Day  30  overlay  trials  (r  =  .377,  p  <  .05). 
These  results  provided  little  evidence  that  differences  in 
psychomotor  skills  (as  measured  by  the  CCHA  cursor  placement 
task)  shared  a  relationship  with  task  performance. 

Psychomotor  skill  retention.  A  graphic  summary  of 
psychomotor  performance  times  is  provided  as  Figure  14.  Summary 
results  of  paired  t-test  (2-tailed)  comparisons  of  psychomotor 
task  performance  times  (in  seconds)  for  the  Overlay  Skilled 
group  are  provided  as  Appendix  W.  The  Overlay  Skilled  sample 
was  selected  so  that  psychomotor  skill  retention  results  would 
be  directly  comparable  to  the  overlay  procedural  skill  retention 
results.  Examining  psychomotor  skill  learning  across  Day  1 
trials,  results  showed  a  significant  decrease  of  approximately 
23  seconds  in  performance  time  from  First  Day  1  (M  =  90,  SD  = 

23)  to  Last  Day  1  trials  (M  =67,  SD  =  7) ,  t(21)  =  5.484,  £  = 
.001,  power  =  1.00,  suggesting  that  significant  learning  was 
taking  place  in  terms  of  reduced  task  performance  time. 

Looking  at  psychomotor  skill  retention  immediately 
following  the  30  day  retention  interval,  the  comparison  of  Last 
Day  1  (M  =  66,  SD  =  7)  and  First  Day  30  trials  (M  =  69, 

SD  =  8)  yielded  an  increase  in  performance  time  of  only  3 
seconds.  With  a  =  .05,  n  =  21,  and  an  effect  size  of  3  seconds, 
power  =  .25.  For  this  small  time  difference  power  is  extremely 
low,  and  no  statistically  significant  differences  between  the 
compared  trials  were  found. 

Investigating  whether  psychomotor  task  performance  times 
showed  a  recovery  of  skill  on  Day  30  trials,  the  comparison 
showed  a  significant  decrease  of  approximately  5  seconds  from 
First  Day  30  (M  =  69,  SD  =  8)  to  Last  Day  30  trials  (M  =  64, 
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SD  =  8)  ,  t  (20)  =  2.861,  p  =  .01,  power  =  .79  suggesting  a 
significant  improvement  or  recovery  of  psychomotor  skill  in  the 
form  of  reduced  task  performance  time. 
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Figure  14.  Psychomotor  performance  times  across  trials. 

Examining  psychomotor  skills  retention  after  practice  on 
Day  30  tasks,  trials  for  Last  Day  1  (M  =  66,  SD  =  7) ,  and  Last 
Day  30  (M  =  64,  SD  =  7)  were  compared.  With  a  =  .05,  n  =  21, 
and  an  effect  size  of  only  2  seconds,  power  =  .19.  For  this 
small  time  difference  power  is  extremely  low,  and  no 
statistically  significant  differences  between  the  compared 
trials  were  found.  Given  the  lack  of  power  for  the  comparison, 
the  results  do  not  provide  evidence  of  a  lack  of  psychomotor 
skill  decay  after  the  30  day  retention  interval. 

In  summary,  the  results  provided  evidence  that  individual 
soldier  psychomotor  task  performance  times  showed  consistency 
across  Day  1  and  Day  30  trials.  The  results  did  not  provide 
evidence  of  any  relationship  between  psychomotor  task  times  and 
I VIS  task  success.  Comparing  psychomotor  tracking  task 
performance  times  across  the  four  trials,  results  provided 
evidence  of  significant  motor  skill  learning  across  Day  1 
trials,  and  significant  skill  recovery  with  practice  on  Day  30 
trials.  The  results  showed  that  the  relative  levels  of 
psychomotor  skill  were  very  similar  at  the  end  of  Day  1  and  Day 
30  trials. 


Issue  6.  Declarative  Knowledge  Assessment 


Declarative  knowledge  scale  reliability.  In  examining  the 
relationship  between  declarative  knowledge  and  IVIS  task 
performance,  it  was  necessary  to  first  establish  the  reliability 
of  the  seven-item  declarative  knowledge  scale.  Here, 
reliability  bears  on  the  interpretation  of  any  relationship 
found  between  declarative  knowledge  scores  and  task  success.  If 
scale  reliability  was  low,  the  relation  could  not  be  high. 
Cronbach's  alpha  for  estimating  scale  reliability  based  on  the 
internal  consistency  of  the  scale  items  was  calculated.  The 
full  sample  of  28  soldiers  was  included  in  the  calculation  to 
provide  the  broadest  possible  range  of  scores.  Cronbach's  alpha 
for  the  declarative  knowledge  scale  was  a  =  .70  for  the  Day  1,  a 
=  .47  for  Start  Day  30,  and  a  =  . 69  for  End  Day  30 
administrations  of  the  questionnaire,  which  suggests  that  the 
scale  has  moderate  internal  reliability.  An  absolute  standard 
for  scale  reliability  has  not  been  set  as  the  utility  of  a  scale 
will  be  a  function  of  both  reliability  and  the  practical 
benefits  associated  with  using  the  scale. 

In  addition  to  Cronbach's  alpha,  test-retest  correlations 
were  computed  to  estimate  the  reliability  of  the  declarative 
knowledge  seven-item  scale  across  Day  1  and  Day  30  evaluations. 
The  correlation  matrix  showing  the  relationship  between  the 
three  declarative  knowledge  evaluation  scores  and  the  sum  of 
successful  overlay  and  report  trials  for  Day  1  and  Day  30 
evaluations  is  provided  as  Appendix  X.  Examining  declarative 
knowledge  test-retest  reliability,  there  was  a  significant 
correlation  between  Day  1  and  First  Day  30  evaluations  (r  =  .43, 
p  <  .05)  and  between  Day  1  and  Last  Day  30  evaluations  (r  =  .60, 
p  <  .001).  These  findings  suggest  that  the  Day  1  declarative 
knowledge  test  seven-item  scale  did  show  evidence  of  significant 
test-retest  reliability  across  Day  1  and  Day  30  trials. 

Declarative  knowledge  scores  and  IVIS  task  success. 

Research  suggests  that  information  stored  as  declarative 
knowledge  is  retained  better  than  simple  procedural  knowledge 
(Druckman  &  Bjork,  1991) .  The  scores  on  the  Day  1  declarative 
knowledge  test  for  the  full  sample  of  28  soldiers  were  first 
compared  to  the  total  number  of  successful  Day  1  overlay  and 
report  trials,  and  were  found  to  be  significantly  related  to 
total  successful  Day  1  overlay  trials  (r  =  .57,  p  <  .001,  n  = 
28),  and  report  trials  (r  =  .59,  p  <  .001,  n  =<28)  (1-tailed 
correlation) .  A  validity  coefficient  was  calculated  to  estimate 
the  strength  of  the  relationship  between  the  paper  and  pencil 
test  of  IVIS  knowledge,  and  hands-on  task  performance  30  days 


39 


later.  Appendix  V  presents  the  full  correlation  matrix  of 
declarative  knowledge  scores  and  overlay  and  report  success.  A 
significant  relationship  was  found  between  Day  1  declarative 
knowledge  scores  and  the  total  number  of  successful  Day  30 
overlay  trials  (r  =  .35,  p  <  .05,  n  =  27),  and  successful  Day  30 
report  trials  (r  =  .67,  p  <  .01,  n  =  29) .  These  results  suggest 
that  the  seven-item  declarative  knowledge  test  shares  something 
in  common  with  factors  contributing  to  Day  30  hands-on  task 
performance . 


Declarative  knowledge  retention.  One  goal  of  the  present 
evaluation  was  to  identify  whether  declarative  knowledge  scores 
would  show  retention  trends  similar  to  procedural  tasks. 
Declarative  knowledge  test  mean  scores  are  presented  graphically 
in  Figure  15.  The  summary  results  of  paired  t-test  comparisons 
of  declarative  knowledge  test  scores  are  provided  as  Appendix  Y . 
The  Overlay  Skilled  sample  was  selected  so  that  declarative 
knowledge  results  and  overlay  task  retention  results  would  be 
based  on  the  same  group  of  soldiers . 


Figure  15.  Declarative  knowledge  scores  comparison. 

As  indicated  in  Appendix  X,  the  decrease  in  test  scores 
from  Day  1  (M  =  6.33,  SD  =  1.06)  to  First  Day  30  (M  =  4.48, 

SD  =  1.25)  evaluations  was  statistically  significant,  t(20)  = 
6.669,  p  <  .001,  power  =  1.00,  and  suggested  that  participants 
did  forget  some  IVIS  overlay  and  reports  task  procedures  as 
measured  by  the  seven-item  questionnaire  over  the  30  day  delay. 
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Examining  declarative  knowledge  skill  decay  after  soldiers 
had  practice  on  the  Day  30  tasks,  the  decrease  in  test  scores 
from  Day  1  (M  =  6.33,  SD  =  1.06)  to  Last  Day  30  (M  =  6.00,  SD  = 
0.89)  was  not  significant,  t(20)  =  1.435.  With  a  =  .05,  n  =  21, 
and  an  effect  size  of  .33,  power  =  .24.  For  this  small 
difference  in  test  scores  power  is  extremely  low,  and  no 
statistically  significant  differences  between  the  compared 
trials  were  found. 

Examining  skill  recovery,  test  scores  improved 
significantly  after  practice  on  the  seven  Day  30  tasks.  The 
difference  between  First  Day  30  (M  =  4.48,  SD  =  1.25)  and  Last 
Day  30  scores  (M  =  6.00,  SD  =  0.89)  was  statistically 
significant,  t(20)  =  4.544,  p  <  .001,  power  =  .99,  and  provided 
evidence  that  soldiers  recovered  IVIS  task  cognitive  skill  with 
practice  during  the  Day  30  trials. 

In  summary,  the  seven-item  declarative  knowledge  multiple 
choice  test  did  show  moderate  internal  reliability.  A 
significant  relationship  was  found  between  Day  1  declarative 
knowledge  scores  and  Day  30  task  success  for  both  IVIS  overlays 
and  reports.  The  results  provided  evidence  of  a  significant 
decay  in  task  knowledge  over  the  thirty  day  retention  interval, 
and  a  significant  recovery  of  this  knowledge  with  practice 
during  Day  30  trials.  The  results  do  not  support  any 
conclusions  regarding  the  relative  levels  of  task  knowledge  at 
the  end  of  Day  1  and  Day  30  to  indicate  whether  any  skill  loss 
remained  after  practice  with  Day  30  trials. 

Demographic  Factors  Evaluation 

The  relationship  between  eight  demographic  measures  and  Day 
30  overlay  and  report  task  success  was  explored  using  Pearson 
correlation  coefficients.  Detailed  results  of  the  evaluation 
are  presented  as  Appendix  Z.  For  this  evaluation,  demographic 
measures  were  compared  to  the  total  number  of  successful  Day  30 
overlay  trials,  and  report  trials.  The  full  population  of  28 
soldiers  was  included  to  provide  the  broadest  possible  range  of 
scores .  Of  the  eight  demographic  factors  evaluated,  only  two 
items  addressing  computer  usage  showed  a  significant 
relationship  with  IVIS  task  performance.  Responses  to  the 
question  "Approximately  how  often  do  you  use  a  computer  each 
month?"  ranged  from  "Never"  to  "Daily"  and  showed  a  significant 
relationship  with  the  total  number  of  successful  Day  30  Overlay 
trials  (r  =  .41,  p  <  .05,  n  =  27) .  Responses  to  the  question 
which  asked  soldiers  to  report  their  experience  with  computers 
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in  terms  of  years  showed  a  significant  relationship  with  the 
total  number  of  successful  Day  30  Report  trials  (r  =  .41, 
p  <  .05,  n  =  21)  . 

Questions  addressing  soldier  motivation  to  learn  tasks  on 
Day  1  and  to  perform  tasks  on  Day  30  were  included  in  the  IVIS 
System  Procedures  Review  questionnaires  to  assess  whether 
soldier  motivation  might  be  a  factor  contributing  to  task 
performance.  On  the  Day  1  questionnaire  all  28  participants 
indicated  that  they  were  either  "Very  Motivated, "  or  "Fairly 
Motivated"  to  learn  IVIS  tasks,  with  no  soldiers  selecting  the 
"Fairly  Unmotivated,"  or  "Very  Unmotivated"  response  options. 

On  the  Day  30  questionnaire  all  27  of  the  remaining  soldiers 
indicated  that  they  were  either  "Very  Motivated, "  or  "Fairly 
Motivated"  to  perform  IVIS  tasks  that  day.  These  results 
provide  some  evidence  that  skill  acquisition  on  Day  1,  and  skill 
retention  Day  30,  were  not  depressed  by  a  lack  of  soldier 
motivation. 


Discussion 


Skill  Retention  Overview 

The  present  research  met  the  goal  of  developing  empirical 
estimates  of  procedural  skill  retention  for  selected  M1A2  IVIS 
digital  communications  tasks.  Results  showed  a  significant  52 
percent  decrease  in  the  proportion  of  soldiers  able  to  meet  the 
Overlay  Skilled  criteria  after  the  30  day  retention  interval 
for  IVIS  overlay  tasks.  Results  also  revealed  a  23  percent 
decrease  in  the  proportion  of  soldiers  meeting  the  Report 
Skilled  criteria.  Psychomotor  skill  did  not  show  significant 
decay  after  the  30  day  retention  interval,  and  also  failed  to 
show  a  relationship  with  overlay  or  report  task  success. 
Declarative  knowledge  of  task  procedures  showed  significant 
decay  after  30  days,  however  this  knowledge  was  recovered  with 
hands-on  experience  during  the  Day  30  evaluation  trials.  The 
present  research  provided  evidence  of  a  significant  relationship 
between  the  seven-item  declarative  knowledge  test  scores  and  Day 
30  overlay  and  report  task  hands-on  performance. 

The  research  design  stressed  that  an  assessment  of  initial 
skill  acquisition  was  a  necessary  prerequisite  to  any 
investigation  of  skill  retention.  In  examining  skill  retention, 
multiple  comparisons  were  employed  to  estimate  Day  1  pro¬ 
ficiency,  Day  30  skill  retention,  and  the  ability  of  soldiers  to 
recover  skills  with  practice  on  Day  30  trials.  An  error  coding 
method  was  developed  which  could  aid  training  developers  in 
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resolving  training  and  equipment  design  issues.  The  error 
coding  method  identifies  the  source  of  performance  problems  at 
the  task,  subtask,  and  individual  task  step  level.  While 
focusing  primarily  on  procedural  aspects  of  task  performance, 
the  research  also  investigated  the  retention  of  declarative 
knowledge,  and  psychomotor  skill,  and  their  relationship  with 
task  success. 

Digital  Procedural  Skill  Acquisition 

Skill  acquisition  is  a  prerequisite  condition  for  any 
assessment  of  skill  retention.  In  the  present  experiment  21 
percent  of  the  soldiers  in  the  full  sample  failed  to  meet  the 
Overlay  Skilled  criteria,  and  18  percent  failed  to  meet  the 
Report  Skilled  criteria  during  Day  1  training  and  evaluation. 

The  scores  for  these  soldiers  were  not  included  in  the 
procedural  skill  retention  estimates.  Failure  to  eliminate  the 
non-skilled  soldiers  scores  would  have  resulted  in  an 
underestimate  of  skill  retention,  reducing  the  proportion  of 
soldiers  meeting  the  Day  30  Overlay  Skilled  criteria  from  48 
percent  to  41  percent.  Likewise,  the  inclusion  of  soldiers  who 
failed  to  attain  the  Report  Skilled  criteria  on  Day  1  would  have 
reduced  the  apparent  Day  30  Report  Skilled  retention  rate  from 
77  percent  to  67  percent.  These  findings  suggest  that  anecdotal 
accounts  of  digital  skill  decay  from  field  trials  might  provide 
an  underestimate  skill  retention  by  failing  to  differentiate 
between  those  soldiers  who  have  lost  skills,  and  those  who  did 
not  initially  acquire  the  skills. 

Digital  Procedural  Skill  Retention 

Measuring  IVIS  skill  retention  at  the  task,  subtask,  and 
individual  task  step  levels  yielded  information  that  could 
greatly  assist  training  developers  in  their  efforts  to  identify 
and  prioritize  training  revision  and  equipment  design 
requirements.  The  IVIS  task  performance  errors  were  treated  as 
misunderstandings,  rather  than  random  mistakes,  and 
systematically  coded  and  quantified  so  that  the  source  of  the 
misunderstanding  could  be  identified. 

Task  level  assessment  of  retention.  Results  of  the  present 
research  revealed  a  significant  52  percent  drop  in  overlay  task 
proficiency  30  days  after  training,  and  a  significant  23  percent 
drop  in  report  task  proficiency.  This  task  level  assessment 
provides  a  useful  summary  of  the  magnitude  of  observed  skill 
loss,  however  it  does  not  identify  the  source  of  performance 
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problems  and  thus  fails  to  provide  a  direction  for  training 
redesign  efforts. 

Subtask  level  assessment  of  retention.  Subtasks  were 
defined  as  clusters  of  separate  actions  which  must  be  performed 
as  a  step-by-step  sequence.  The  identification  of  subtasks  for 
IVIS  overlay  and  report  tasks  advances  the  measure  of  skill 
retention  beyond  a  simple  task  success  rate.  In  the  present 
research,  subtask  retention  assessment  identified  IVIS  overlay 
TRP  and  SEND  subtasks  as  contributing  most  to  the  observed  Day 
30  reduction  in  task  success.  Likewise,  the  failure  to  retain 
IVIS  report  ACTION  and  SEND  subtask  skills  was  identified  as  the 
primary  source  of  Day  30  report  skill  loss.  These  subtask  level 
skill  retention  results  might  suggest  to  training  developers  the 
need  to  investigate  why  soldiers  failed  to  complete  the  steps 
necessary  to  send  a  message  for  both  overlay  and  report  tasks. 

Task  Step  Level  Assessment  of  Retention.  The  collection 
and  analysis  of  task  step  performance  data  was  extremely  useful 
in  identifying  the  source  of  performance  errors.  Task  step 
coding  provided  the  quantitative  data  necessary  to  identify 
specific  recurring  errors  where  soldiers  repeatedly  selected  an 
incorrect  (but  often  logical)  response  alternative.  The  task 
step  analysis  also  helped  to  identify  the  location  in  the  IVIS 
lesson  plan  where  emphasis  might  be  placed  to  correct 
misunderstandings,  or  where  essential  information  could  be 
added.  The  present  research  identified  a  recurring  error  where 
soldiers  would  select  the  “SEND"  option  and  exit  the  task,  not 
realizing  that  the  "SEND"  option  does  not  complete  the  action  of 
sending  an  overlay  or  report.  The  detailed  step  level  analysis 
also  revealed  that  training  materials  failed  to  provide  a 
procedure  for  correcting  target  data  entry  mistakes  in  the  Spot 
Report.  These  results  illustrate  the  value  of  recording  task 
step  performance  data  to  facilitate  training  and  training  system 
development . 

Psychomotor  Skill  Assessment 

The  present  research  investigated  the  retention  of  IVIS 
related  psychomotor  skill,  and  its  relationship  with  digital 
procedural  task  performance.  Results  of  the  research  suggested 
that  psychomotor  skills  associated  with  the  IVIS  CCHA  interface 
did  not  show  significant  decay  after  the  30  day  retention 
interval.  Likewise,  results  provided  little  evidence  that 
psychomotor  skill  shared  a  relationship  with  overlay  and  report 
task  performance. 
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One  goal  in  exploring  psychomotor  skill  was  to  demonstrate 
how  this  type  of  information  could  assist  training  developers  in 
prioritizing  training  system  requirements.  The  CCHA  hand 
controller  provided  with  the  CST  is  an  expensive  full-fidelity 
training  tool.  With  IVIS  software  now  being  hosted  on  desktop 
personal  computers,  the  cost  of  the  CCHA  can  equal  the  cost  of 
the  remaining  components  in  the  CST  training  system.  Given  the 
minimal  learning  and  absence  of  decay  associated  with  IVIS 
psychomotor  skill,  the  present  research  raised  the  issue  of 
whether  the  expense  of  a  full  fidelity  CCHA  hand  controller  was 
justified. 

Declarative  Knowledge  Skill  Assessment 

The  present  research  investigated  the  retention  of 
declarative  knowledge  of  IVIS  task  procedures,  and  suggested 
training  requirements  associated  with  acquiring  and  maintaining 
this  skill.  Declarative  knowledge  of  task  procedures  showed 
significant  decay  after  30  days,  however  this  knowledge  was 
recovered  with  unaided  hands-on  experience  during  the  Day  30 
evaluation  trials.  Results  revealed  a  significant  relationship 
between  the  seven- item  declarative  knowledge  Day  1  test  scores 
and  Day  30  overlay  and  report  task  hands-on  performance. 

The  significant  relationship  found  between  IVIS  task 
procedure  declarative  knowledge  and  hands-on  performance  has 
important  implications  for  the  way  training  developers  design 
and  deliver  course  instruction.  The  NETT  approach  to  IVIS 
training  could  be  characterized  as  training  to  a  level  of 
procedural  knowledge,  where  soldiers  followed  the  instructors 
verbal  instructions  to  perform  overlay  and  report  sequential 
task  steps  on  CSTs.  Written  training  materials  were  not 
provided,  and  little  emphasis  was  placed  on  providing  a  "big 
picture"  summary  of  the  IVIS  menu  structure  and  menu 
organization  principles.  Training  to  this  procedural  level  of 
knowledge  could  leave  the  soldier  capable  of  performing  overlay 
and  report  tasks,  but  unable  to  describe,  or  remember,  what  task 
steps  were  performed,  or  why. 

Training  to  a  declarative  level  of  knowledge  has  the 
potential  to  enhance  digital  procedural  skill  retention  and 
should  be  considered  in  training  development  efforts.  For  IVIS 
this  training  approach  could  be  characterized  as  including 
instruction  and  evaluation  on  the  organizing  concepts  that 
underlie  the  digital  system  menu  structure.  Here,  training 
could  be  designed  to  present  soldiers  with  evaluation  tasks  that 
require  them  to  consolidate  their  hands-on  knowledge  of  task 
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procedures  into  clear  statements  describing  how  tasks  are 
performed. 

Task  Characteristics  Supporting  Skill  Retention 

In  searching  for  a  valid  approach  to  investigate  skill 
retention  differences  across  tasks,  the  User's  Decision  Aid 
skill  retention  rating  scale  developed  by  Rose  et  al.,  (1985) 
was  found  to  provide  a  useful  framework  for  comparing  tasks  on  a 
common  set  of  ten  retention  related  factors. 

Predicting  skill  retention.  The  User's  Decision  Aid  was 
first  employed  in  the  present  experiment  to  provide  skill 
retention  estimates  for  overlay  and  report  tasks.  The 
comparison  of  observed  (48%)  to  predicted  (67%)  overlay  skill 
retention  was  not  significant,  while  the  comparison  of  observed 
(77%)  to  predicted  (92%)  report  skill  retention  was  significant. 
The  results  of  this  comparison  were  mixed,  with  the  report  task 
comparison  showing  a  significant  difference  between  observed  and 
predicted  skill  retention.  For  both  overlays  and  reports  the 
rating  method  tended  to  under-predict  retention  loss. 

Overlay  and  report  retention  factor  comparison.  Results  of 
the  present  research  do  not  directly  address  the  question  of  why 
overlay  tasks  appear  to  show  greater  skill  loss  (52%)  than  do 
report  tasks  (23%)  after  the  30  day  retention  interval.  It 
should  be  noted  that  a  statistical  comparison  of  retention  rates 
is  not  appropriate  as  the  overlay  and  report  tasks  differ  with 
regard  to  skill  acquisition  and  retention  criteria,  numbers  of 
tasks  presented,  and  order  of  presentation.  Likewise,  a 
comparison  of  error  trends  across  tasks  has  little  utility  as 
the  subtasks  and  task  steps  for  overlays  and  reports  are  very- 
different. 

The  User's  Decision  Aid  ten  rating  factors  provide  a 
valuable  framework  for  comparing  overlay  and  report  task 
requirements  associated  with  skill  retention.  Appendix  S 
presents  the  ten  retention  factor  ratings  for  the  overlay  and 
report  tasks.  The  ratings  indicate  that  the  overlay  and  report 
tasks  differ  in  terms  of  (1)  job  or  memory  aid  quality,  (2) 
number  of  facts  to  remember,  (3)  how  hard  facts  and  terms  are  to 
remember,  and  (4)  motor  control  demands  of  the  task.  Comparing 
overlay  and  report  tasks  on  the  rating  factors,  the  greater 
skill  decay  observed  for  overlay  tasks  might  be  a  function  of 
the  greater  flexibility  required  to  create  a  variety  of  map 
graphics,  which  limits  opportunities  for  job  aiding  in  the  form 
of  common  language  or  single  function  menu  labels.  In  contrast. 
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the  highly  structured  format  of  reports  allows  for  menu  label 
job  aids  that  guide  the  soldier  through  a  f ill-in-the-blank  task 
step  sequence .  The  greater  flexibility  in  overlay  task 
requirements  may  also  require  the  soldier  to  commit  more  menu 
rules  to  memory  rather  than  providing  this  information  as  menu 
label  cues.  With  regard  to  motor  control  demands,  the  overlay 
experiment  tasks  presented  a  greater  number  of  precision  motor 
skill  requirements  compared  to  the  report  task.  For  overlay 
tasks  soldiers  were  required  to  place  two  points  on  each  overlay 
map  with  +/-  10  meter  precision,  compared  to  only  one  target 
location  with  +  /-  10  meter  precision  for  report  tasks. 

Future  Directions 


Rapid  change  is  an  inherent  characteristic  of  software- 
driven  digital  communication  systems  such  as  the  IVIS.  Future 
training  research  should  explore  opportunities  to  shift  away 
from  the  short-term  goal  of  training  single-system  procedural 
tasks,  to  a  long-term  training  method  that  prepares  soldiers 
with  a  knowledge  of  digital  system  operating  concepts.  This 
concept  level  knowledge  of  system  operation  could  facilitate  the 
long-term  retention  and  transfer  of  skills  across  changing 
software  and  equipment.  Here,  the  increased  front-end  training 
costs  might  be  recovered  by  reducing  the  requirement  for 
multiple  single-system  training  programs  across  the  duration  of 
a  soldiers  career. 
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Appendix  A 


List  of  Acronyms 

AFRU  .  Armored  Forces  Research  Unit 

ARI . Army  Research  Institute  for  the  Behavioral  and 

Social  Sciences 

AWE  .  Advanced  Warfighting  Experiment 

CCHA  .  Commander's  Control  Handle  Assembly 

CST . Crew  Station  Trainer 

CVC2  .  Combat  Vehicle  Command  and  Control 

FASTTRAIN  .  .  Force  XXI  Training  Methods  and  Strategies 

IVIS  .  Inter -Vehicular  Information  System 

MEDEVAC.  .  .  .  Medical  Evacuation 

MGRS . Military  Grid  Reference  System 

MOS  .  Military  Occupational  Specialty 

NETT . New  Equipment  Training  Team 

OPTEC  ....  Operational  Test  and  Evaluation  Command 
SINCGARS  .  .  .  Single  Channel  Ground/Airborne  Radio  System 

SOP  .  Standard  Operating  Procedure 

SME  .  Subject  Matter  Expert 

TRP  .  Target  Reference  Point 
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Video  Data  Reduction  Sheets 


MISSION  PLANNING  VIDEO  REDUCTION 

TASK  NUMBER  1  TASK  SEQUENCE  _  PIN 

1.  Start  Time  (Press  Pre/Post  or  Combat)  _ 

2.  Stop  Time  (Press  CONFIRM  RT  A)  _ 

3 .  Total  Time  _ 

4.  100%  accurate  info  sent? 


Task  steps: _ 

YES  NO  ERRORS 

_  _  _  1.  Press  PRE/POST  or  COMBAT  (Start  NAV:  _ ) 

_  _  _  2.  Press  MISSION  PLANNING  . 

_  _  _  3.  Press  EDIT  OVERLAY 

_  _  _  4.  OBSTACLE  overlay 

_  _  _  5.  Press  GRAPHICS 

_  _  _  6.  press  PAGE  DOWN /PAGE  UP 

_  _  _  7.  MINEFIELD  graphic  (Start  GRAPH:  _ ) 

_  _  _  8.  CURSOR  used 

_  _  _  9.  Coordinates  2400  2550  and  2400  2500 

_  _  _  10.  Press  RETURN 

_  _  _  11.  TRP  selected  (Start  TRP:  _ ) 

_  _  _  12.  KEYPAD  used  to  enter  2300  2750 

_  _  _  13.  Coordinate  2300  2750 

_  _  _  14 .  Press  RETURN 

_  _  _  15.  Press  LABEL 

_  _  _  16.  Enter  text  AA0001 

_  _  _  17.  Text  entered  on  2ND  line 

_  _  _  18.  Press  RETURN  button  (Start  SEND:  _ ) 

_  _  _  19.  Press  SAVE  CHANGES 

_  _  _  20.  Press  SEND  OVERLAY 

_  _  _  21.  Select  OBSTACLE  overlay 

_  _  _  22.  Select  STANDARD  ROUTING 

_  _  _  23.  Press  CONFIRM  RT  A 

24.  Comments  code  number 


COMMENTS: 
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REPORTS  VIDEO  REDUCTION 


TASK  NUMBER  2  TASK  SEQUENCE  _  PIN 

1.  Start  Time  (Press  Pre/Post  or  Combat) 

2.  Stop  Time  (Press  CONFIRM  RT  A)  _ 

3 .  Total  Time  _ 

4.  100%  accurate  information  sent? 


Task  steps: _ 

YES  NO  ERRORS 

_  _  _  1. 

_  _  _  2. 

_  _  _  3. 

_  _  _  4. 

_  _  _  5. 

_  _  _  6. 

_  _  _  7. 

_  _  _  8. 

_  _  _  9. 

_  _  _  10. 

_  _  _  11. 

_  _  _  12 . 

_  _  _  13. 

_  14. 

_  15. 

_  _  _  16. 

_  17. 

_  _  _  18. 

_  _  _  19. 

20. 


Press  COMBAT  (Start  NAV:  _ ) 

Press  REPORTS 
Press  IVIS  REPORTS 

SPOT  REPORT  Selected  (Start  GRID:  _ ) 

2650  2500  Target  coord 

CURSOR  used  to  enter  target  grid 

APC  Selected  (Start  TGTS:  _ ) 

BMP  Selected 
(2) Enter  size 
TANK  Selected 
T72  Selected 
(4)  Enter  size 

ACTIVITY  Selected  (Start  ACTIVITY:  _ ) 

WITHDRAW  Selected 
FRIENDLY  ACTION  Selected 
CONTINUE  Selected 

Press  SEND  (Start  SEND:  _ ) 

STANDARD  ROUTING 
Press  CONFIRM  RT  A 
Comments  code  number 


Comments : 
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Appendix  C 


IVIS  SYSTEM  PROCEDURES  REVIEW  1 

PIN:  _ 


Please  answer  each  question  below  by  circling  a  letter. 

1.  What  mode  button  would  you  select  first  to  edit  an  IVIS 
OVERLAY? 

_ 2  a.  IVIS  Reports 

26  *b.  Pre/Post  or  Combat 

_ 0  c .  Diag 

_ 0  d.  IVIS 

_ 0  e.  Map  Tools 

2.  What  mode  would  you  choose  first  to  begin  sending  an  IVIS 
REPORT? 

_ 1  a.  Pre/Post 

21  *b.  Combat 

_ 1  c .  Diag 

_ 5  d.  Mission  Planning 

_ 0  e.  Map  Tools 

3 .  Which  pushbutton  would  you  select  to  center  the  IVIS  map 
around  your  vehicle? 

11  a.  Scroll  Off /On 
16  *b.  Scroll  Home 

_ 1  c .  Zoom 

_ 0  d.  Location  Off/On 

_ 0  e.  Vehicle  Iden  Off /On 

4.  Which  pushbutton  would  you  select  to  add  graphics  to  an 
existing  overlay? 

_ 0  a.  Map  Tools 

27  b.  Graphics 

_ 0  c.  Delete 

_ 0  d.  Decltr 

1  e .  Label 


5.  Which  pushbutton  would  you  select  to  temporarily  hide 
selected  graphics? 

_ 2  a.  Map  Tools 

_ 2  b.  Graphics 

_ 0  c.  Delete 

_ 2  *d.  Decltr 

_ 0  e .  Label 

6.  How  many  characters  of  text  can  be  entered  in  one  line  using 


the 

FREE 

TEXT  feature? 

0 

a. 

Four 

0 

b. 

Six 

1 

c . 

Eight 

27 

*d. 

Ten 

0 

e. 

Fifteen 

7.  When  using  the  alpha  numeric  keypad,  if  the  data  entry  point 
appears  as  a  highlighted  box  (not  a  highlighted  underline)  what 
will  the  keypad  record? 

_ 2  a.  A  number 

26  *b.  A  letter 

8.  When  using  the  alpha  numeric  keypad,  does  the  cursor 
automatically  advance  one  space  when  you  enter  a  number? 

22  *a.  Yes 
_ 6  b.  No 

9.  Circle  the  letter  of  the  correct  sequence  you  would  follow 
to  create  a  Contact  Report. 

27  *a.  COMBAT  -  REPORTS  -  I VIS  REPORTS  -  CONTACT  REPORT 

1  b.  PRE/POST  -  MISSION  PLANNING  -  REPORTS  -  CONTACT  REPORT 

_ 0  C.  COMBAT  -  COMMO  -  IVIS  -  CONTACT  REPORT 

_ 0  d.  PRE/POST  -  MISSION  PLANNING  -  COMMO  -  CONTACT  REPORT 

10.  Circle  the  letter  of  the  correct  sequence  you  would  follow 
to  add  a  graphic  to  the  Enemy  overlay. 

3  a.  COMBAT  -  IVIS  -  EDIT  OVERLAY  -  ENEMY 
20  *b.  PRE/POST  -  MISSION  PLANNING  -  EDIT  OVERLAY  -  ENEMY 
5  c.  COMBAT  -  MISSION  PLANNING  -  DISPLAY  OVERLAY  -  ENEMY 

0  d.  PRE/POST  -  IVIS  -  DISPLAY  OVERLAY  -  ENEMY 
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11.  How  likely  is  it  that  you  will  be  able  to  quickly  and 
accurately  create  and  send  an  IVIS  map  OVERLAY  one  month  from 
today  without  additional  practice? 

9  a.  Very  likely 
19  b.  Fairly  likely 

_ 0  c.  Fairly  unlikely 

_ 0  d.  Very  unlikely 

12.  How  likely  is  it  that  you  will  be  able  to  quickly  and 
accurately  create  and  send  an  IVIS  REPORT  one  month  from  today 
without  additional  practice? 

11  a.  Very  likely 
16  b.  Fairly  likely 

_ 1  c.  Fairly  unlikely 

_ 0  d.  Very  unlikely 

13.  How  likely  is  it  that  you  will  have  to  perform  computer- 
based  tasks  in  your  current  or  next  assignment? 

_ 5  a.  Very  likely 

18  b.  Fairly  likely 

_ 5  c.  Fairly  unlikely 

_ 0  d.  Very  unlikely 

14.  How  adequately  did  the  CST  training  prepare  you  to  perform 
OVERLAY  tasks? 

25  a.  Very  adequate 

_ 3  b.  Fairly  adequate 

_ 0  c.  Fairly  inadequate 

_ 0  d.  Very  inadequate 

15.  How  adequately  did  the  CST  training  prepare  you  to  perform 
REPORT  tasks? 

26  a.  Very  adequate 

_ 2  b.  Fairly  adequate 

_ 0  c.  Fairly  inadequate 

_ 0  d.  Very  inadequate 

*  Correct  response  option  for  items  1-10. 
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PT  60-15 

IVIS  SYSTEM  PROCEDURES  REVIEW  2 


PIN: 


Please  answer  each  of  the  following1  questions  by  circling  a 
letter. 

1.  What  mode  button  would  you  select  first  to  edit  an  IVIS 
OVERLAY? 

_ 3  a.  IVIS  Reports 

16  *b.  Pre/Post  or  Combat 
_ 4  c .  Diag 

_ 2  d.  IVIS 

_ 2  e.  Map  Tools 

2 .  What  mode  would  you  choose  first  to  begin  sending  an  IVIS 
REPORT? 

_ 2  a.  Pre/Post 

17  *b .  Combat 
_ 4  c.  Diag 

_ 4  d.  Mission  Planning 

_ 0  e.  Map  Tools 

3 .  Which  pushbutton  would  you  select  to  center  the  IVIS  map 
around  your  vehicle? 

_ 6  a.  Scroll  Off/On 

16  *b.  Scroll  Home 

_ 0  c .  Zoom 

_ 4  d.  Location  Off/On 

_ 1  e.  Vehicle  Iden  Off /On 

4.  Which  pushbutton  would  you  select  to  add  graphics  to  an 
existing  overlay? 


4 

a. 

Map  Tools 

20 

*b. 

Graphics 

0 

c. 

Delete 

0 

d. 

Decltr 

3 

e. 

Label 
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5.  Which  pushbutton  would  you  select  to  temporarily  hide 
selected  graphics? 


1 

a. 

Map  Tools 

1 

b. 

Graphics 

2 

c . 

Delete 

23 

*cL 

Decltr 

0 

e. 

Label 

6. 

How  many  characters  of  text  can 

be  entered  in  one  line 

using 

the 

FREE 

TEXT  feature? 

3 

a. 

Four 

2 

b. 

Six 

7 

c. 

Eight 

13 

*d. 

Ten 

2 

e. 

Fifteen 

7. 

When 

using  the  alpha  numeric  keypad,  if  the  data  entry 

point 

appears  as  a  highlighted  box  (not  a 

highlighted  underline) 

what 

will  the  keypad  record? 

9 

a. 

A  number 

18 

*b. 

A  letter 

8. 

When 

using  the  keypad,  does  the 

cursor  automatically  advance 

one 

space  when  you  enter  a  number? 

18 

*a. 

Yes 

9 

b. 

No 

9.  Circle  the  letter  of  the  correct  sequence  you  would  follow 
to  create  a  Contact  Report. 

22  *a.  COMBAT  -  REPORTS  -  IVIS  REPORTS  -  CONTACT  REPORT 
3  b.  PRE/POST  -  MISSION  PLANNING  -  REPORTS  -  CONTACT  REPORT 
1  c.  COMBAT  -  COMMO  -  IVIS  -  CONTACT  REPORT 
0  d.  PRE/POST  -  MISSION  PLANNING  -  COMMO  -  CONTACT  REPORT 

10.  Circle  the  letter  of  the  correct  sequence  you  would  follow 
to  add  a  graphic  to  the  Enemy  overlay. 

9  a.  COMBAT  -  IVIS  -  EDIT  OVERLAY  -  ENEMY 
7  *b.  PRE/POST  -  MISSION  PLANNING  -  EDIT  OVERLAY  -  ENEMY 
TO  C.  COMBAT  -  MISSION  PLANNING  -  DISPLAY  OVERLAY  -  ENEMY 
1  d.  PRE/POST  -  IVIS  -  DISPLAY  OVERLAY  -  ENEMY 

11.  How  motivated  are  you  to  perform  the  IVIS  tasks  in  the 
evaluation  sessions  today? 

18  a.  Very  motivated 
9  b.  Fairly  motivated 
0  c.  Fairly  unmotivated 
0  d.  Very  unmotivate 
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12.  How  likely  is  it  that  you  will  be  able  to  quickly  and 
accurately  create  and  send  an  IVIS  map  OVERLAY  today? 

_ 5  a.  Very  likely 

19  b.  Fairly  likely 
2  c.  Fairly  unlikely 
1  d.  Very  unlikely 


13.  How  likely  is  it  that  you  will  be  able  to  quickly  and 
accurately  create  and  send  an  IVIS  REPORT  today? 

_ 5  a.  Very  likely 

18  b.  Fairly  likely 

_ 2  c.  Fairly  unlikely 

_ 1  d.  Very  unlikely 

14.  How  likely  is  it  that  you  will  have  to  perform  computer- 
based  tasks  in  your  current  or  next  assignment? 

_ 8  a.  Very  likely 

14  b.  Fairly  likely 

_ 4  c.  Fairly  unlikely 

_ 0  d.  Very  unlikely 

15.  How  adequately  did  the  CST  training  prepare  you  to  perform 
OVERLAY  tasks? 

18  a.  Very  adequate 

_ 7  b.  Fairly  adequate 

_ 1  c.  Fairly  inadequate 

_ 0  d.  Very  inadequate 

16.  How  adequately  did  the  CST  training  prepare  you  to  perform 
REPORT  tasks? 

18  a.  Very  adequate 

_ 7  b.  Fairly  adequate 

_ 1  c.  Fairly  inadequate 

_ 0  d.  Very  inadequate 

17.  Since  you  completed  IVIS  OVERLAYS  training  on  the  Crew 
Station  Trainer  30  days  ago,  have  you  practiced  creating  an  IVIS 
overlay? 

_  (describe  below) 

Number  of  times  practiced  _ 

Number  of  total  hours  practiced  _ 

Number  of  days  since  last  practiced  _ 


2_6  No 
0  Yes 
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18.  Since  you  completed  IVIS  REPORTS  training  on  the  Crew 
Station  Trainer  30  days  ago,  have  you  practiced  creating  an 
IVIS  report? 

26  No  _ 

_ 0  Yes  _  (describe  below) 

Number  of  times  practiced  _ 

Number  of  total  hours  practiced  _ 

Number  of  days  since  last  practiced  _ 

*  Correct  response  option  for  items  1-10. 
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PT  60-15 


IVIS  SYSTEM  PROCEDURES  REVIEW  3 


PIN: 


Please  answer  each  of  the  following  questions  by  circling  a 
letter. 

1.  What  mode  button  would  you  select  first  to  edit  an  IVIS 
OVERLAY? 

_ 1  a.  IVIS  Reports 

23  *b.  Pre/Post  or  Combat 
_ 1  c.  Diag 

_ 0  d.  IVIS 

_ 2  e.  Map  Tools 

2.  What  mode  would  you  choose  first  to  begin  sending  an  IVIS 
REPORT? 

_ 0  a.  Pre/Post 

22  *b.  Combat 

_ 0  c .  Diag 

_ 5  d.  Mission  Planning 

_ 0  e.  Map  Tools 

3 .  Which  pushbutton  would  you  select  to  center  the  IVIS  map 
around  your  vehicle? 

_ 8  a.  Scroll  Off/On 

16  *b.  Scroll  Home 

_ 0  c .  Zoom 

_ 1  d.  Location  Off/On 

_ 2  e.  Vehicle  Iden  Off /On 

4 .  Which  pushbutton  would  you  select  to  add  graphics  to  an 
existing  overlay? 

_ 1  a.  Map  Tools 

24  *b.  Graphics 

_ 1  c.  Delete 

_ 0  d.  Decltr 

1  e.  Label 
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5.  Which  pushbutton  would  you  select  to  temporarily  hide 
selected  graphics? 

_ 1  a.  Map  Tools 

_ 0  b.  Graphics 

_ 1  c.  Delete 

25  *d.  Decltr 
_ 0  e .  Label 

6.  How  many  characters  of  text  can  be  entered  in  one  line  using 


the 

FREE 

TEXT  feature? 

3 

a. 

Four 

1 

b. 

Six 

9 

c. 

Eight 

12 

*d. 

Ten 

2 

e. 

Fifteen 

7 .  When  using  the  alpha  numeric  keypad,  if  the  data  entry  point 
appears  as  a  highlighted  box  (not  a  highlighted  underline)  what 
will  the  keypad  record? 

_ 2  a.  A  number 

25  *b.  A  letter 

8.  When  using  the  keypad,  does  the  cursor  automatically  advance 
one  space  when  you  enter  a  number? 

24  *a.  Yes 
_ 3  b.  No 

9.  Circle  the  letter  of  the  correct  sequence  you  would  follow 
to  create  a  Contact  Report. 

26  *a.  COMBAT  -  REPORTS  -  IVIS  REPORTS  -  CONTACT  REPORT 

_ 1  b.  PRE/POST  -  MISSION  PLANNING  -  REPORTS  -  CONTACT  REPORT 

0  c.  COMBAT  -  COMMO  -  IVIS  -  CONTACT  REPORT 
0  d.  PRE/POST  -  MISSION  PLANNING  -  COMMO  -  CONTACT  REPORT 

10.  Circle  the  letter  of  the  correct  sequence  you  would  follow 
to  add  a  graphic  to  the  Enemy  overlay. 

4  a.  COMBAT  -  IVIS  -  EDIT  OVERLAY  -  ENEMY 
16  *b.  PRE/POST  -  MISSION  PLANNING  -  EDIT  OVERLAY  -  ENEMY 
7  c.  COMBAT  -  MISSION  PLANNING  -  DISPLAY  OVERLAY  -  ENEMY 

_ 0  d.  PRE/POST  -  IVIS  -  DISPLAY  OVERLAY  -  ENEMY 

11.  How  adequately  did  the  CST  training  prepare  you  to  perform 
OVERLAY  tasks? 

19  a.  Very  adequate 

_ 8  b.  Fairly  adequate 

_ 0  c.  Fairly  inadequate 

0  d.  Very  inadequate 
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12'.  How  adequately  did  the  CST  training  prepare  you  to  perform 
REPORT  tasks? 

19  a.  Very  adequate 

_ 8  b.  Fairly  adequate 

0  c.  Fairly  inadequate 
_ 0  d.  Very  inadequate 

*  Correct  response  option  for  items  1-10. 
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Overlay  Task  Elements 


Task 

Number 

Overlay 

Multiple  Point 
Graphic 

Graphic 

Label 

1 

Obstacle 

Minefield 

AA0001 

3 

Obstacle 

Anti-Tank  Ditch 

AA0002 

5 

Operations 

1 

Bridge 

AA0005 

7 

Enemy 

Free  Draw 

AA0004 

8 

Obstacle 

Minefield 

AA0003 

10 

Obstacle 

Anti-tank  Ditch 

AA0006 

12 

Operations 

1 

Bridge 

AA0007 

14 

Enemy 

Free  Draw 

AA0008 

D-l 
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Report  Task  Elements 


Task 

Number 

TGT  1/ 

Subtype /No . 

TGT  2/ 
Subtype /No . 

Friendly 

Action 

Enemy 

Activity 

2 

APC/BMP/2 

TANK/T72/4 

Continue 

Withdraw 

4 

APC/BRDM/4 

TANK/T80/2 

Observing 

Reconn 

6 

ARTILLERY/SP/6 

APC/BTR/2 

Observing 

Defending 

9 

APC/BMP/4 

TANK/T72/2 

Continue 

Withdraw 

11 

APC/BRDM/2 

TANK/T80/4 

Observing 

Re conn 

13 

ARTILLERY/ SP/ 4 

APC/BTR/4 

Observing 

Defending 

Appendix  F 


Essential  IVIS  Task  Success  Criteria 


IVIS  Overlay  and  Report  Data  Elements  Required  for  Success 


Essential  IVIS  overlay  information  elements 

1.  Select  the  correct  IVIS  Overlay. 

2.  Select  the  correct  multi-point  graphic. 

3 .  Place  the  multi-point  graphic  at  the  correct  location 
(+/-  100  meters) . 

4.  Place  a  TRP  at  the  correct  location  (requires  exact 
coordinates  entered  with  keypad) . 

5.  Label  the  TRP  (either  left  or  right  side)  to  include  the 
correct  two  letters  and  four  digits  (ex.  AA0006) . 

6.  Send  the  Overlay  over  RT  A  or  RT  A/B. 

Essential  IVIS  report  information  elements 

1.  Select  the  correct  IVIS  Report  type  (Spot  Report) 

2.  Place  the  target  icon  graphic  on  the  tactical  display  at 
the  correct  location  (+/-  100  meters) 

3 .  Enter  the  specified  Target  1  type 

4.  Enter  the  specified  Target  1  subtype 

5.  Enter  the  specified  Target  1  size 

6.  Enter  the  specified  Target  2  type 

7.  Enter  the  specified  Target  2  subtype 

8.  Enter  the  specified  Target  2  size 

9.  Enter  the  specified  Type  of  Enemy  Activity 

10.  Enter  the  specified  Type  of  Friendly  Action 

11.  Send  the  Overlay  over  RT  A  or  RT  A/B. 


Note.  Entering  extra  targets  with  a  size  other  than  zero 
constitutes  failure.  Sending  a  second  report  over  the  same  net 
constitutes  failure  as  second  report  overwrites  first  with 
a  blank  report . 


F-l 


APPENDIX  G  GOES  HERE 
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Example:  16S  ET2500  2600 


Thumb  cursor  standard  =  +/- 10  Example:  2550  =  2540  to  2560 
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Psychomotor  Ten  Digit  Task 


1 _ 1 _ _ 1 

, 

■ 

10 

1 

1 

8 

3 

6 

5 

4 

7 

2 

9 

Note.  Soldiers  would  use  CCHA  thumb  cursor  to  click  on  and 
highlight  numbers  in  numerical  sequence  beginning  with  number  1 
and  ending  with  number  10. 
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Appendix  I 

Demographic  Data  Survey 


PT  60-15 


DIGITAL  SKILL  RETENTION 
DEMOGRAPHIC  SURVEY 

NAME: _ _  SOCIAL  SECURITY  NUMBER: 

PIN: _ 


1.  BC/MOS:  _  2.  Rank/Grade:  _  3.  Age:  _ 

4.  Time  in  Military  Service: 

Officer:  Years  _  Months  _ 

Enlisted:  Years  _  Months  _ 

5.  Circle  your  current  duty  position/assignment: 

Driver  Loader  Gunner  Tank  Cdr  Pit  Sgt  Pit  Ldr  Co  XO  Co  Cdr 
Other :  (please  describe) _ 

6.  Have  you  trained  and/or  had  experience  with  any  digital 
equipment  used  in  the  M1A2  tank  or  similar  systems?  Please  list 
by  type  (i.e.,  SINCGARS,  IVIS,  Applique,  B2C2,  ASAS) ,  training 
received  and/or  experience,  and  dates. 


7.  If  you  related  your  experience  with  computers  in  terms  of 
years,  where  would  you  place  yourself  on  the  chart  below? 
(Circle  a  number  please.) 

1  2  3  4  5 

No  less  than  more  than  more  than  5  years 

experience  1  year  1  year  3  years  or  more 
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8.  Approximately  how  often  do  you  use  a  computer  each  month? 
(Circle  a  number  please.) 

1  2  3  4  5 


Never  Once  a  Once  a  2-3  times  Daily 
month  week  per  week 


9.  Do  you  use  Windows  (or  a  Windows-type  software  [Win  3.11, 
Win  95,  MAC])  while  using  a  computer? 

_  Yes 

_  No 

10.  Please  circle  the  value  that  best  describes  how  you  feel 
(in  general)  about  using  computers. 


Very  Uncomfortable  Neutral  Comfortable  Very 
Uncomfortable  Comfortable 

11.  Please  check  your  highest  civilian  education  level 
achieved: 


High  school  diploma/GED 

Some  college 

College  degree  (BA/BS) 

Graduate-level  courses  (no  graduate  degree) 
Graduate  degree  (MA/MS) 
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Summary  of  Demographic  Data 


1. 

BC/MOS:  All 

are  19K  qualified 

2. 

Rank/Grade : 

7  E-2  4  E-3 

12 

E-4 

5_ 

E-5 

3. 

Average  age: 

22.29  years 

4. 

Average  time 

in  military  service? 

2 

years 

11 

months 

5. 

Duty  Position. 

13  Driver 

2  Loader  6  Gunner 

4 

Tank  Cdr 

3 

Other 

6.  Digital  equipment  training 

13  None  15  SINCGARS  0  IVIS 

6.  Experience  with  computers,  years  of  experience. 

6  No  Experience  ljL  Less  Than  1  Year  _4  More  Than  1  Year 
5  More  Than  3  Years  _2  5  Years  Or  More 

7.  How  often  do  you  use  a  computer  each  month? 

8  Never  12  Once  A  Month  _3  Once  A  Week 

2  2-3  Times  Per  Week  _3  Daily 

8.  Do  you  use  Windows  while  using  a  computer?  19  Yes  _9  No 

9.  Describe  how  you  feel  about  using  computers? 

1  Very  Uncomfortable  _1  Uncomfortable  __9  Neutral 
12  Comfortable  _5  Very  Comfortable 

10.  Highest  civilian  education  level. 

14  High  School  Diploma  13  Some  College 
1  College  Degree  (BA/BS) 
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Overlay  Trials  Success  and  Time  Summary 


Overlay  Trial  Success  Rate  Performance  Time  (sec) 


M 

SD 

Day 

1  Session 

(n  =  28) 

Trial 

1  (First 

Day  1) 

.68 

219.64 

59.20 

Trial 

3 

.61 

228.57 

67.71 

Trial 

5 

.79 

194.39 

56.11 

Trial 

7  (Last  Day  1) 

.74 

188.63 

51.75 

Day 

30  Session 

(n  =  27) 

Trial 

8  (First 

Day  30) 

.22 

464.00 

214.40 

Trial 

10 

.41 

351.50 

212.43 

Trial 

12 

.44 

258.81 

168.59 

Trial 

14  (Last 

Day  30) 

.41 

214.62 

83.28 

Note.  Time  data  includes  unsuccessful  trials.  One  overlay 
performance  time  value  for  Trial  5  (406  seconds)  was  identified 
as  falling  more  than  three  standard  deviations  beyond  the  mean 
for  that  particular  trial .  Following  the  data  trimming  approach 
outlined  in  Marascuilo  and  McSweeney  (1977)  this  value  was 
replaced  with  the  next  lowest  time  value  and  the  lowest  time 
value  was  replaced  with  the  next  highest  value. 
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Report  Trials  Success  and  Time  Summary 


Report 

.  Trial 

Success  Rate 

Performance  Time  (sec) 

M  SD 

Day 

1  Session 

(n  =  28) 

Trial 

2  (First 

Day 

1) 

.71 

165.04 

58.90 

Trial 

4 

.74 

126.74 

44.00 

Trial 

6  (Last  Day  1) 

.82 

109.64 

26.95 

Day 

30  Session 

(n  =  27) 

Trial 

9  (First 

Day 

30) 

.67 

214.04 

143.02 

Trial 

11 

.63 

127.37 

41.49 

Trial 

13  (Last 

Day 

30) 

.70 

128.37 

83.37 

Note.  Time  data  includes  unsuccessful  trials.  Four  report 
performance  time  values  were  identified  as  falling  more  than 
three  standard  deviations  beyond  the  mean  for  that  particular 
trial.  Trial  2  (555),  Trial  4  (281),  Trial  9  (865),  and  Trial  11 
(697)  .  Following  the  data  trimming  approach  outlined  in 
Marascuilo  and  McSweeney  (1977)  each  value  was  replaced 
with  the  next  lowest  time  value  and  the  lowest  time  value 
was  replaced  with  the  next  highest  value. 
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Overlay  Trials  Summary  For  Skill  Acquired  Group 


Overlay  Trial 

Success 

Rate 

Rate 

n 

Performance 
M  SD 

Time  (sec) 
n 

DAY  1  SESSION 

Trial  1  (First 

Day  1) 

.82 

22 

213.11 

56.97 

18 

Trial  3 

.73 

22 

210.81 

59.45 

16 

Trial  5 

.95 

22 

186.48 

54.27 

21 

Trial  7  (Last  Day  1) 

.86 

22 

180.26 

44.64 

19 

DAY  30  SESSION 

Trial  8  (First 

Day  30) 

.29 

21 

509.00 

199.57 

6 

Trial  10 

.48 

21 

275.70 

82.92 

10 

Trial  12 

.52 

21 

202.82 

44.29 

11 

Trial  14  (Last 

Day  30) 

.48 

21 

183.20 

60.96 

10 

Note.  Table  includes  only  successful  trial  times. 
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Report  Trials  Summary  for  Skill  Acquired  Group 


Report  Trial 

Success 

Rate 

Rate 

n 

Performance  Time 
M  SD 

(sec) 

n 

DAY  1  SESSION 

Trial  2  (First  Day  1) 

.83 

23 

146.84 

36.95 

19 

Trial  4 

.91 

22 

125.15 

44.03 

20 

Trial  6  (Last  Day  1) 

.91 

23 

106.24 

23.07 

21 

DAY  30  SESSION 

Trial  9  (First  Day  30) 

.77 

22 

180.71 

117.24 

17 

Trial  11 

.68 

22 

115.53 

25.93 

15 

Trial  13  (Last  Day  30) 

.77 

22 

116.65 

43.94 

17 

Note.  Table  includes  only  successful  trial  times. 


N-l 


Appendix  0 

Overlay  Task  Success  Comparisons 


Trial  Contrast 


Trial  Success  Comparisons 


Proportion  Difference  Sig. 
Successful 


Learning : 


First  Day  1  vs. 

.82 

.040 

.500 

Last  Day  1 

.86 

Retention  1:  . 

•  ■ 

Last  Day  1  vs. 

.86 

-.570 

.001** 

First  Day  30 

.29 

Retention  2 : 

Last  Day  1  vs. 

.86 

-.380 

.029* 

Last  Day  30 

.48 

Recovery : 

First  Day  30  vs. 

.29 

.190 

.109 

Last  Day  30 

.48 

Overlay  Skilled 

Day  1  vs. 

1.00 

.520 

.000** 

Day  30 

.48 

Note.  McNemar  test  of  related  proportions. 

exact 

significance 

(1-tailed) .  n  =  22  for 

Day 

1  comparisons . 

n  =  : 

21  for 

comparisons  that  include 

Day 

30  trials. 

*p  <  .05.  **p  <  .001. 
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Appendix  P 

Report  Task  Success  Comparisons 


Trial  Contrast 

Trial 

Success  Comparisons 

Proportion 

Successful 

Difference 

Sig. 

Learning : 

First  Day  1  vs. 

.83 

.080 

.344 

Last  Day  1 

.91 

Retention  1 : 

Last  Day  1  vs. 

.91 

-.140 

.188 

First  Day  30 

.77 

Retention  2 : 

Last  Day  1  vs. 

.91 

-.140 

.188 

Last  Day  30 

.77 

Recovery : 

First  Day  30  vs. 

.77 

.000 

.688 

Last  Day  30 

.77 

Report  Skilled 

Day  1  vs. 

1.00 

.230 

.003* 

Day  30 

.77 

Note.  McNemar  test  of  related  proportions,  exact  significance 
(1-tailed).  n  =  23  for  Day  1  comparisons,  n  =  22  for 
comparisons  that  include  Day  30  trials. 

*£  <  . 05 . 
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Overlay  Task  Time  Comparisons 


Trial  Contrast 

M 

SD 

Dif  f . 

t/df 

Sig. 

Learning : 


First  Day  1  vs. 
Last  Day  1 

213.80 

179.33 

61.97 

35.69 

-34.47 

2.994/14 

.010** 

Retention  1: 

Last  Day  1  vs. 
First  Day  30 

196.00 

518.00 

47.65 

221.77 

-321.40 

3.861/4 

.018* 

Retention  2 : 

Last  Day  1  vs. 
Last  Day  30 

177.86 

164.38 

51.26 

31.07 

-17.71 

1.723/6 

.136 

Recovery: 

First  Day  30  vs. 
Last  Day  30 

491.40 

179.17 

217.86 

30.09 

-315.20 

3.688/4 

.021* 

Note.  Paired  sample  t-test  (2-tailed) . 
*p  <  .05.  **  p  <  .01. 
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Report  Task  Time  Comparisons 


Trial  Contrast 

M 

SD 

Diff . 

t/df 

Sig. 

Learning: 

First  Day  1  vs. 
Last  Day  1 

147.65 

101.41 

39.05 

18.23 

-46.24 

6.208/16 

.000* 

Retention  1 : 

Last  Day  1  vs. 
First  Day  30 

101.94 

154.31 

24.84 

45.05 

52.38 

5.383/15 

.000* 

Retention  2: 

Last  Day  1  vs. 
Last  Day  30 

103.75 

118.75 

24.72 

44.49 

15.00 

1.327/15 

.204 

Recovery: 

First  Day  30  vs. 
Last  Day  30 

186.73 

115.40 

124.00 

46.38 

-71.33 

2.020/14 

.063 

Note.  Paired  sample  t-test  (2-tailed) . 

*p  <  .001. 
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Skill  Retention  Ratings 


Scale  Question 


1.  Job/Memory  Aid 

Yes  =  1 

No  =  0  (Go  to  3) 

2 .  Job /Memory  Aid  Quality 

Excellent  =  56  (Go  to  6) 
Very  Good  =25 
Marginally  Good  =  2 
Poor  =  1 

3 .  Number  of  Steps 

1  Step  =  25  (Go  to  6) 

2  to  5  Steps  =  14 
6  to  10  Steps  =  12 
More  than  10  =  0 

4 .  Sequence 

None  Are  =  10 
All  Are  =  5 

Some  Are,  Some  Are  Not  = 

5 .  Feedback 

For  All  Steps  =  22 
For  Most  Steps  =  19 
Only  a  Few  =  11 
None  =  0 

6 .  Time 

None  =  40 
Easy  =  35 
Difficult  =  0 

7 .  Mental  Requirements 

Almost  None  =  37 
Simple  =  28 
Complex  =  3 
Very  Complex  =  0 


Overlay  Task  Report  Task 


1  1 


25 

1 


0  0 

0  0  0 

19  19 

40  40 

37  37 
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Retention  Scale  Ratings  (continued) 


Scale  Question 


Overlay  Task  Report  Task 


8 .  Number  of  Facts 

None  =20 
A  Few  (1-3)  =  18 
Some  (4-8)  =  13 

More  than  8)  =  0  0 

9 .  How  Hard  to  Remember 

Not  Applicable  =34 
Not  Hard  At  All  =  31 
Somewhat  Hard  =  12  12 

Very  Hard  =  0 

10.  Motor  Control  Requirements 

None  =  2 
Small  =  0 

Considerable  =  16  16 

Very  Great  =  3 


13 


31 


0 


Total  Score  126  166 

Estimated  Retention 

After  30  Days  67%  92% 
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Subtask  Step  Errors  and  Recommendations 


Summary  of  Frequent  Overlay  Subtask  Step  Errors  (All  Trials) 


Subtask 

Task  Step  Error 

Description/Recommendation 

Navigate  to 
Overlay 

104:  Select 
Overlay 

an 

Fails  to  recognize  interface 
requirement  to  highlight  an 
overlay  from  menu. 

Recommend :  Increase 
training  emphasis,  add 
"Select  an  Overlay"  message 
when  Overlay  menu  displayed. 

Multiple 

Point 

Graphic 

204:  Grid  points 
wrong 

Fails  to  remember  how  to 
plot  simple  MGRS  coords . 

Recommend:  Increase  MGRS 

training  emphasis? 

TRP  Graphic 

303:  Cursor  used 
to  enter  coord 

Fails  to  remember  that 
cursor  is  not  accurate 
enough  for  TRP  coord. 

Recommend:  Add  illustration 
examples  comparing  cursor 
and  keypad  accuracy. 

Send 

Overlay 

405:  Select 
Overlay 

an 

Fails  to  recognize  interface 
requirement  to  highlight  an 
overlay  from  menu. 

Recommend:  Increase 
training  emphasis,  add 
"Select  an  Overlay"  message 
when  Overlay  menu  displayed. 

Summary  of  Frequent  Report  Subtask  Step  Errors  (All  Trials) 


Subtask 

Task  Step  Error 

Description/Recommendation 

Navigate  to 
Report 

No  errors 

No  errors 

Enter  Target 
Grid 

601:  Grid  points 
wrong 

Fails  to  remember  how  to 
plot  simple  MGRS  coords . 

Recommend:  May  be  limited 
to  experiment,  normally 
lazing  inserts  target  icon. 

Target 

Description 

709:  Bad  data 
from  unsent 
report  remains 

Failure  to  send  previous 
report,  and  can't  overwrite 
old  data  with  new  data. 

Recommend:  No  data  overwrite 
procedure  in  Lesson  Plan. 
Create  one  and  train  it. 
Change  software  to  dump  data 
after  exiting  report  mode? 

Friendly/ 

Enemy 

Activity 

806:  Selects 

DIAG  mode  after 
entering  data 

Ends  the  task  after  entering 
Report  data,  forgets  to  send 
the  report. 

Recommend:  May  be  limited 
to  experiment .  Graphics 
mode  prompts  operator  to 
SAVE,  Report  mode  software 
could  prompt  to  SEND. 

Send  Report 

908:  DIAG  mode 
after  SEND 

Fails  to  select  CONFIRM  RT  A 
after  selecting  SEND  option. 

Recommend:  Train  that  SEND 
does  not  actually  send. 
Report  mode  software  could 
prompt  to  CONFIRM  RT. 
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Overlay  and  Report  Task  Step  Errors 


Frequency  of  Overlay  Day  1  and  Day  30  Task  Step  Errors 


Code 

Step  Number  and  Description 

Day  1 

Day  30 

Total 

100 

NAVIGATE  TO  OVERLAY  GRAPHICS 

2 

14 

16 

Select  PRE/POST 


Select  MISSION  PLANNING 


Select  EDIT  OVERLAY 


Select  correct  overlay 


Select  GRAPHICS 


PAGE  DOWN/UP  to  select  graphic 


Coaching  Required 


200 

TWO- 

-POINT  GRAPHIC 

201 

7. 

Select  two-point  graphic  (wrong  one) 

202 

Select  two-point  graphic  (no  select) 

203 

8. 

Use  cursor  for  grid  points 

204 

9. 

Both  grid  points  correct 

205 

Extra  two-point  graphic  applied 

206 

10. 

Press  RETURN  to  exit  graphic 

207 

Selects  "CONTINUE- 

208 

Selects  " DIAG "  after  two-point 

TRP  GRAPHIC 


11.  Select  TRP  graphic  (wrong  graphic) 


302 

Select  TRP  graphic  (none  selected) 

3 

5 

8 

303 

12 .  Use  keypad  to  enter  TRP 

10 

14 

24 

304 

13 .  TRP  grid  point  correct 

3 

3 

Select  RETURN  (not  required) 


Select  LABEL 


TRP  label  content  correct 


TRP  labeled  on  left  side 


Two-point  graphic  labeled 


310 

18.  Select  RETURN  (exit  TRP  menu) 

4 

4 

311 

Extra  graphic  applied 

5 

10 

15 

312 

Label  text  accurate 

2 

6 

8 

313 

TRP  Deleted 

1 

1 

2 

314 

Selects  DIAG  after  TRP  labeled 

3 

5 

8 
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Frequency  of  Overlay  Task  Step  Errors  (continued) 

fOD  I  SAVE  AND  SEND  OVERLAY  ' 


19.  Select  SAVE  CHANGES  _ 


Saves  by  selecting  a  MODE _ 


_ Saves  by  going  to  SAVE  prompt _ 

20.  Select  SEND  OVERLAY _ 


Selects  DIAG  after  SEND  OVERLAY 


21.  Select  overlay  to  send  (no  select) 


Select  overlay  to  send  (wrong  one) 


22.  Select  STANDARD  ROUTING  _ 


23 .  Confirm  on  RT  A  _ 


Selects  RT  A/B  _ 


Selects  RT  B  _ 


Duplicate  overlay (s)  sent _ 


Select  DIAG  _ 


DIAG  selected  after  SAVE  _ 


No  SAVE/SEND  actions 


Frequency  of  Report  Task  Step  Errors 

Code  I  Step  Number  and  Description 


Day  1  I  Day  30  |  Total 


500 

Navigate  to  Spot  Report 

501 

1.  Select  COMBAT 

502 

2 .  Press  REPORTS 

503 

3.  Select  IVIS  REPORTS 

504 

4.  Select  SPOT  REPORTS 

Enter  Target  Grid _ 

5 .  Enter  correct  target  grid  coord 


6.  Cursor  used  to  enter  target  grid 


Target  grid  not  entered 


Enter  Target  Description  Data _ 


7.  Select  first  target  type  _ 


8.  Select  first  target  sub- type _ 


9.  Enter  first  target  size  with  keypad 


10 .  Select  second  target  type _ 

11.  Select  second  target  sub- type 


12 .  Enter  second  target  size  with  keypad 


Target  data  not  entered _ 


Only  one  target  type  entered _ 


Tgt.  data  from  unsent  report  remains 


Selects  DIAG  after  entering  tgt  data 


Target  data  entered  twice _ 


Can't  eliminate  bad  target  data 


12 

23 

4 

5 

5 

14 

3 

4 

Frequency  of  Report  Task  Step  Errors  (continued) 


800 

Enter  Friendly  Actions /Enemy  Activity 

10 

14 

24 

801 

13.  Select  ACTIVITY 

1 

1 

2 

802 

14.  Select  activity  type 

1 

1 

mm 

15.  Select  FRIENDLY  ACTION 

1 

4 

5 

mm 

16.  Select  action  type 

2 

2 

805 

This  code  not  used 

806 

Selects  DIAG  after  friend/enemy  data 

3 

5 

8 

807 

No  friendly  or  enemy  info  entered 

3 

3 

6 

900 

Send  Report 

4 

25 

29 

901 

17.  Select  SEND 

1 

2 

3 

mm 

18.  Select  STD  ROUTING 

903 

19.  Select  CONFIRM  RT  A 

904 

Select  CONFIRM  RT  A/B 

10 

10 

905 

Select  CONFIRM  RT  B 

2 

2 

906 

Duplicate  report  sent  (blank) 

2 

2 

4 

907 

Select  DIAG 

908 

Go  directly  to  DIAG  after  SEND 

4 

4 

909 

Code  not  used 

910 

No  SEND  actions  taken 

1 

5 

6 

911 

Technical  problem  (cursor  drift) 

Note.  Errors  are  for  full  sample  of  soldiers,  not  restricted  to 
the  Skill  Acquisition  groups,  n  =  28.  Each  soldier  may 
contribute  more  than  one  error  per  subtask  per  trial. 
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Intercorrelations  Between  Psychomotor  Trial  (Motor) 


Time,  and  Successful 

Overlay  and 

Report 

Trials 

Measure  123 

4 

5 

6 

7 

8 

1.  Motor  1  --  .59**  .64** 

.45** 

.00 

.18 

.25 

.27 

2.  Motor  2  --  .34* 

.30 

.02 

-.09 

.16 

.26 

3.  Motor  3  (Day  30) 

.57** 

-.13 

.18 

.30 

.32 

4.  Motor  4  (Day  30) 

-.01 

.38* 

.12 

.24 

5.  Overlay  Total  (Day  1) 

— 

.44* 

.47** 

.31 

6.  Overlay  Total  (Day  30) 

— 

.48** 

.41* 

7.  Report  Total  (Day  1) 

-- 

.48** 

8.  Report  Total  (Day  30) 

-- 

Note,  n  =  28  for  correlations  (1-tailed)  based  exclusively  on 
Day  1  scores,  n  =  27  for  correlations  (1-tailed)  that  include 
Day  30  scores. 

*p  <  .05. .  **£  <  .01. 
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Psychomotor  Test  Time  Comparisons 


Trial  Contrast 

M 

SD 

Diff . 

t/df 

Sig. 

Learning : 

First  Day  1  vs. 

90 

23 

23.45 

5.484/21 

.000** 

Last  Day  1 

67 

7 

Retention  1: 

Last  Day  1  vs. 

66 

7 

-2.57 

1.352/20 

.191 

First  Day  30 

69 

9 

Retention  2: 

Last  Day  1  vs . 

66 

7 

2.29 

1.139/20 

.268 

Last  Day  30 

64 

7 

Recovery : 

First  Day  30  vs. 

69 

8 

4.86 

2.861/20 

.010* 

Last  Day  30 

64 

8 

Note.  Sample  is  restricted  to  Overlay  Skilled  group. 
Comparisons  are  paired  t-tests  (2-tailed) . 

*p  <  .01.  **£  <  .001. 
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Correlations  Between  Declarative 
Knowledge  (DK)  Scores  and  Task  Success 


Measure  1  2  3  4  5  6 


1. 

Day  1  DK  —  .43* 

.60** 

.57** 

.35* 

.59** 

.68** 

2. 

First  Day  30  DK  — 

.14 

.15 

.35* 

.42* 

.43* 

3. 

Last  Day  30  DK 

— 

.17 

.06 

.29 

.35* 

4. 

Overlay  Day  1  Total 

— 

.44* 

.47** 

.31 

5. 

Overlay  Day  30  Total 

— 

.48** 

.41* 

6. 

Report  Day  1  Total 

-- 

.48** 

7. 

Report  Day  30  Total 

— 

Note,  n  =  28  for  correlations  (1-tailed)  based  exclusively  on 
Day  1  scores,  n  =  27  for  correlations  (1-tailed)  that  include 
Day  30  scores. 

*g  <  .05. .  **p  <  .01. 
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Appendix  Y 


Declarative  Knowledge  Test  Comparisons 


Trial  Contrast 

M 

SD 

Diff . 

t/df 

Sig. 

Retention  1 : 

Last  Day  1  vs. 
First  Day  30 

6.33 

4.84 

1.06 

1.25 

-1.86 

6.669/20 

.000* 

Retention  2 : 

Last  Day  1  vs. 

Last  Day  30 

6.33 

6.00 

1.06 

0.89 

-.33 

1.435/20 

.167 

Recovery: 

First  Day  30  vs. 
Last  Day  30 

4.48 

6.00 

1.25 

0.89 

1.52 

4.544/20 

.000* 

Note.  Sample  is  restricted  to  Overlay  Skilled  group. 
Comparisons  are  paired  sample  t-tests  (2-tailed) . 

*p  <  .001. 
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Appendix  Z 


Correlations  Between  Demographic 
Characteristics  and  Day  30  Trial  Success 


Correlation  With  Day  30  Total 
of  Successful  Trials  (r/sig.) 

Demographic  Factor 

Overlays 

Reports 

Rank  (E2  -  E5) 

.114/. 287 

.000/1.000 

Age  (20  -  29) 

.084/. 679 

.015/. 943 

Duty  Position 
(1  =  Tank  CDR,  0  =  other) 

.232/. 244 

.176/. 381 

Time  in  service  (months) 

(9  -  120) 

.147/. 465 

.178/. 357 

Highest  education  level 
(HS  Degree  -  College) 

-.165/. 410 

.051/. 802 

Use  computer  each  month 
(Never  -  Daily) 

.411/. 033* 

.110/. 584 

Computer  experience 
(None  -  5  yrs  or  more) 

.088/. 663 

.407/. 035* 

Do  you  use  Windows  (yes /no) 

-.204/. 308 

-.242/. 224 

Note,  n  =  27  due  to  transfer  of  one  soldier. 
*p  <  .05  (2-tailed) . 
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